[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:46:32 dist-test-slave-kvfs krb5kdc[9538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:46:32 dist-test-slave-kvfs krb5kdc[9538](info): set up 2 sockets
Aug 10 01:46:32 dist-test-slave-kvfs krb5kdc[9538](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 01:46:34.379784 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.457s user 0.002s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:46:34 dist-test-slave-kvfs krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:46:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:46:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:46:34.536707 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:42697
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:40629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:42697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:46:34.672588 9555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:46:34.672961 9555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:46:34.673044 9555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:46:34.677156 9555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:46:34.677244 9555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:46:34.677281 9555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:46:34.677320 9555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:46:34.677352 9555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:46:34.682823 9555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:40629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:42697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:42697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:46:34.684247 9555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:46:34.685530 9555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:46:34.696172 9560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:34.696211 9563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:34.696175 9561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:34.696223 9555 server_base.cc:1034] running on GCE node
I20260810 01:46:34.697359 9555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:46:34.698781 9555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:46:34.699991 9555 hybrid_clock.cc:648] HybridClock initialized: now 1786326394699935 us; error 74 us; skew 500 ppm
Aug 10 01:46:34 dist-test-slave-kvfs krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326394, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:46:34.703500 9555 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:46:34.704725 9555 webserver.cc:460] Webserver started at http://127.9.77.190:42111/ using document root <none> and password file <none>
I20260810 01:46:34.705408 9555 fs_manager.cc:359] Metadata directory not provided
I20260810 01:46:34.705474 9555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:46:34.705763 9555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:46:34.708443 9555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "8f03f0f59f00426ca16af134b00b64f2"
format_stamp: "Formatted at 2026-08-10 01:46:34 on dist-test-slave-kvfs"
server_key: "573ff6250d4399045902a0c53b7db232"
server_key_iv: "90faf007a213bcab9f2f4ac22850f088"
server_key_version: "encryptionkey@0"
I20260810 01:46:34.709076 9555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "8f03f0f59f00426ca16af134b00b64f2"
format_stamp: "Formatted at 2026-08-10 01:46:34 on dist-test-slave-kvfs"
server_key: "573ff6250d4399045902a0c53b7db232"
server_key_iv: "90faf007a213bcab9f2f4ac22850f088"
server_key_version: "encryptionkey@0"
I20260810 01:46:34.713795 9555 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 01:46:34.716892 9571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:34.718473 9555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 01:46:34.718640 9555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "8f03f0f59f00426ca16af134b00b64f2"
format_stamp: "Formatted at 2026-08-10 01:46:34 on dist-test-slave-kvfs"
server_key: "573ff6250d4399045902a0c53b7db232"
server_key_iv: "90faf007a213bcab9f2f4ac22850f088"
server_key_version: "encryptionkey@0"
I20260810 01:46:34.718784 9555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:46:34.733997 9555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:46:34.737787 9555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:46:34.738030 9555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:46:34.746711 9555 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:42697
I20260810 01:46:34.746728 9656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:42697 every 8 connection(s)
I20260810 01:46:34.747944 9555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:46:34.751588 9658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:34.757308 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 9555
I20260810 01:46:34.757516 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:46:34.759224 9658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f03f0f59f00426ca16af134b00b64f2: Bootstrap starting.
I20260810 01:46:34.759469 9526 external_mini_cluster.cc:1428] Setting key 7d15dc0f2769b32e73288aef11579818
I20260810 01:46:34.762912 9658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f03f0f59f00426ca16af134b00b64f2: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:34.763893 9658 log.cc:824] T 00000000000000000000000000000000 P 8f03f0f59f00426ca16af134b00b64f2: Log is configured to *not* fsync() on all Append() calls
I20260810 01:46:34.766614 9658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f03f0f59f00426ca16af134b00b64f2: No bootstrap required, opened a new log
I20260810 01:46:34.771919 9658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f03f0f59f00426ca16af134b00b64f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f03f0f59f00426ca16af134b00b64f2" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 42697 } }
I20260810 01:46:34.772290 9658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f03f0f59f00426ca16af134b00b64f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:34.772368 9658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f03f0f59f00426ca16af134b00b64f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f03f0f59f00426ca16af134b00b64f2, State: Initialized, Role: FOLLOWER
I20260810 01:46:34.772967 9658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f03f0f59f00426ca16af134b00b64f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f03f0f59f00426ca16af134b00b64f2" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 42697 } }
I20260810 01:46:34.773157 9658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f03f0f59f00426ca16af134b00b64f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:46:34.773237 9658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f03f0f59f00426ca16af134b00b64f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:46:34.773353 9658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f03f0f59f00426ca16af134b00b64f2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:34.774813 9658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f03f0f59f00426ca16af134b00b64f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f03f0f59f00426ca16af134b00b64f2" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 42697 } }
I20260810 01:46:34.775274 9658 leader_election.cc:302] T 00000000000000000000000000000000 P 8f03f0f59f00426ca16af134b00b64f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f03f0f59f00426ca16af134b00b64f2; no voters:
I20260810 01:46:34.775702 9658 leader_election.cc:288] T 00000000000000000000000000000000 P 8f03f0f59f00426ca16af134b00b64f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:46:34.775826 9665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f03f0f59f00426ca16af134b00b64f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:46:34.776082 9665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f03f0f59f00426ca16af134b00b64f2 [term 1 LEADER]: Becoming Leader. State: Replica: 8f03f0f59f00426ca16af134b00b64f2, State: Running, Role: LEADER
I20260810 01:46:34.776463 9665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f03f0f59f00426ca16af134b00b64f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f03f0f59f00426ca16af134b00b64f2" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 42697 } }
I20260810 01:46:34.777292 9658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f03f0f59f00426ca16af134b00b64f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:46:34.778517 9666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f03f0f59f00426ca16af134b00b64f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f03f0f59f00426ca16af134b00b64f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f03f0f59f00426ca16af134b00b64f2" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 42697 } } }
I20260810 01:46:34.778685 9666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f03f0f59f00426ca16af134b00b64f2 [sys.catalog]: This master's current role is: LEADER
I20260810 01:46:34.778998 9667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f03f0f59f00426ca16af134b00b64f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f03f0f59f00426ca16af134b00b64f2. Latest consensus state: current_term: 1 leader_uuid: "8f03f0f59f00426ca16af134b00b64f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f03f0f59f00426ca16af134b00b64f2" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 42697 } } }
I20260810 01:46:34.779093 9667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f03f0f59f00426ca16af134b00b64f2 [sys.catalog]: This master's current role is: LEADER
I20260810 01:46:34.779381 9676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:46:34.783202 9676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 10 01:46:34 dist-test-slave-kvfs krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:34.790581 9676 catalog_manager.cc:1347] Generated new cluster ID: 2a78b4332a664290af56dae3bef7afd2
I20260810 01:46:34.790657 9676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:46:34.793406 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:34.763795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58354 (local address 127.9.77.190:42697)
0810 01:46:34.764397 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:34.764427 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:34.765887 (+ 1460us) server_negotiation.cc:407] Connection header received
0810 01:46:34.771420 (+ 5533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:34.771440 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:34.771816 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:34.772223 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:34.775606 (+ 3383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:34.777329 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:34.779421 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:34.780277 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:34.780473 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:34.786545 (+ 6072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:34.786553 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:34.786574 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:34.786607 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:34.788916 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:34.789713 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:34.789720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:34.789726 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:34.789899 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:34.790347 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:34.790351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:34.790352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:34.791032 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:34.791277 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:34.791945 (+ 668us) server_negotiation.cc:299] Negotiation successful
0810 01:46:34.792277 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":200,"threads_started":1}
I20260810 01:46:34.809195 9676 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:46:34.810372 9676 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:46:34.816672 9676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f03f0f59f00426ca16af134b00b64f2: Generated new TSK 0
I20260810 01:46:34.817657 9676 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.77.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
I20260810 01:46:34.867210 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:0
--local_ip_for_outbound_sockets=127.9.77.129
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:42697
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:40629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:46:35.041145 9697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:46:35.041523 9697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:46:35.041575 9697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:46:35.045922 9697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:46:35.045998 9697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:46:35.046095 9697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:46:35.051893 9697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:40629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:42697
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:46:35.053332 9697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:46:35.054673 9697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:46:35.067159 9705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:35.067111 9707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:35.067128 9704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:35.067767 9697 server_base.cc:1034] running on GCE node
I20260810 01:46:35.068460 9697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:46:35.069373 9697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:46:35.070602 9697 hybrid_clock.cc:648] HybridClock initialized: now 1786326395070575 us; error 39 us; skew 500 ppm
Aug 10 01:46:35 dist-test-slave-kvfs krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326395, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:46:35.074610 9697 init.cc:377] Logged in from keytab as kudu/127.9.77.129@KRBTEST.COM (short username kudu)
I20260810 01:46:35.075749 9697 webserver.cc:460] Webserver started at http://127.9.77.129:34837/ using document root <none> and password file <none>
I20260810 01:46:35.076536 9697 fs_manager.cc:359] Metadata directory not provided
I20260810 01:46:35.076610 9697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:46:35.076835 9697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:46:35.079890 9697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-0/data/instance:
uuid: "dc6214ce8a4348c9bf875cc4e4dff504"
format_stamp: "Formatted at 2026-08-10 01:46:35 on dist-test-slave-kvfs"
server_key: "dc35957524133c5ebf170d505da14a44"
server_key_iv: "a28991e099bae315940fc9d5ac30ee48"
server_key_version: "encryptionkey@0"
I20260810 01:46:35.080714 9697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance:
uuid: "dc6214ce8a4348c9bf875cc4e4dff504"
format_stamp: "Formatted at 2026-08-10 01:46:35 on dist-test-slave-kvfs"
server_key: "dc35957524133c5ebf170d505da14a44"
server_key_iv: "a28991e099bae315940fc9d5ac30ee48"
server_key_version: "encryptionkey@0"
I20260810 01:46:35.085561 9697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 01:46:35.088771 9716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:35.090504 9697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:46:35.090670 9697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "dc6214ce8a4348c9bf875cc4e4dff504"
format_stamp: "Formatted at 2026-08-10 01:46:35 on dist-test-slave-kvfs"
server_key: "dc35957524133c5ebf170d505da14a44"
server_key_iv: "a28991e099bae315940fc9d5ac30ee48"
server_key_version: "encryptionkey@0"
I20260810 01:46:35.090812 9697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:46:35.120527 9697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:46:35.125352 9697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:46:35.125633 9697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:46:35.126441 9697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:46:35.127593 9697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:46:35.127677 9697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:35.127792 9697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:46:35.127851 9697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:35.140266 9697 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:33025
I20260810 01:46:35.140304 9887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:33025 every 8 connection(s)
I20260810 01:46:35.141440 9697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:46:35.148550 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 9697
I20260810 01:46:35.148669 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance
I20260810 01:46:35.149021 9526 external_mini_cluster.cc:1428] Setting key f61fbf5f0e391674953d277a778b606e
Aug 10 01:46:35 dist-test-slave-kvfs krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326395, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:35.159080 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:35.143937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:38831 (local address 127.9.77.190:42697)
0810 01:46:35.144146 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:35.144150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:35.145229 (+ 1079us) server_negotiation.cc:407] Connection header received
0810 01:46:35.146833 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:35.146837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:35.146920 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:35.147013 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:35.148545 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:35.149316 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:35.151021 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:35.151300 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:35.151424 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:35.154376 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:35.154381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:35.154385 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:35.154436 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:35.156337 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:35.157032 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:35.157035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:35.157037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:35.157121 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:35.157548 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:35.157551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:35.157552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:35.157775 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:35.157858 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:35.158701 (+ 843us) server_negotiation.cc:299] Negotiation successful
0810 01:46:35.158878 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 01:46:35.160691 9891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:35.144424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:42697 (local address 127.9.77.129:38831)
0810 01:46:35.145019 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:35.145069 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:35.146577 (+ 1508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:35.147235 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:35.147242 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:35.147690 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:46:35.148370 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:35.148387 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:35.149454 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:35.149464 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:35.150905 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:35.150916 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:35.151466 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:35.151470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:35.151678 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:35.152468 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:46:35.152486 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:46:35.154171 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:46:35.156464 (+ 2293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:35.156474 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:35.156497 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:35.156922 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:35.157213 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:35.157219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:35.157224 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:35.157447 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:35.157893 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:46:35.157904 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:46:35.158353 (+ 449us) client_negotiation.cc:769] Sending connection context
0810 01:46:35.158612 (+ 259us) client_negotiation.cc:240] Negotiation successful
0810 01:46:35.159117 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":149,"threads_started":1}
I20260810 01:46:35.162359 9888 heartbeater.cc:347] Connected to a master server at 127.9.77.190:42697
I20260810 01:46:35.162685 9888 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.9.77.130@KRBTEST.COM; defaulting to no policy
I20260810 01:46:35.163429 9888 heartbeater.cc:511] Master 127.9.77.190:42697 requested a full tablet report, sending...
I20260810 01:46:35.165491 9602 ts_manager.cc:194] Registered new tserver with Master: dc6214ce8a4348c9bf875cc4e4dff504 (127.9.77.129:33025)
I20260810 01:46:35.166905 9602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.129@KRBTEST.COM'} at 127.9.77.129:38831
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.130@KRBTEST.COM" created.
I20260810 01:46:36.170065 9888 heartbeater.cc:503] Master 127.9.77.190:42697 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
W20260810 01:46:36.378597 9526 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.9.77.130: real 1.227s user 0.001s sys 0.005s
I20260810 01:46:36.379484 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:0
--local_ip_for_outbound_sockets=127.9.77.130
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:42697
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:40629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:46:36.517714 9894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:46:36.518182 9894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:46:36.518270 9894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:46:36.522634 9894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:46:36.522738 9894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:46:36.522835 9894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:46:36.528765 9894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:40629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:42697
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:46:36.530187 9894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:46:36.531522 9894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:46:36.543542 9900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:36.543542 9904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:36.543861 9894 server_base.cc:1034] running on GCE node
W20260810 01:46:36.543547 9902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:36.544498 9894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:46:36.545332 9894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:46:36.546532 9894 hybrid_clock.cc:648] HybridClock initialized: now 1786326396546494 us; error 47 us; skew 500 ppm
Aug 10 01:46:36 dist-test-slave-kvfs krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326396, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:46:36.550190 9894 init.cc:377] Logged in from keytab as kudu/127.9.77.130@KRBTEST.COM (short username kudu)
I20260810 01:46:36.551354 9894 webserver.cc:460] Webserver started at http://127.9.77.130:33229/ using document root <none> and password file <none>
I20260810 01:46:36.552158 9894 fs_manager.cc:359] Metadata directory not provided
I20260810 01:46:36.552258 9894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:46:36.552538 9894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:46:36.555119 9894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-1/data/instance:
uuid: "419ed6fa8ae04406828c40f33be38569"
format_stamp: "Formatted at 2026-08-10 01:46:36 on dist-test-slave-kvfs"
server_key: "9637f90e9fd2b157e15234d5b899336a"
server_key_iv: "8a627c5d93875c5df521f2952d198c30"
server_key_version: "encryptionkey@0"
I20260810 01:46:36.555807 9894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance:
uuid: "419ed6fa8ae04406828c40f33be38569"
format_stamp: "Formatted at 2026-08-10 01:46:36 on dist-test-slave-kvfs"
server_key: "9637f90e9fd2b157e15234d5b899336a"
server_key_iv: "8a627c5d93875c5df521f2952d198c30"
server_key_version: "encryptionkey@0"
I20260810 01:46:36.560704 9894 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260810 01:46:36.564852 9917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:36.567049 9894 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 01:46:36.567216 9894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "419ed6fa8ae04406828c40f33be38569"
format_stamp: "Formatted at 2026-08-10 01:46:36 on dist-test-slave-kvfs"
server_key: "9637f90e9fd2b157e15234d5b899336a"
server_key_iv: "8a627c5d93875c5df521f2952d198c30"
server_key_version: "encryptionkey@0"
I20260810 01:46:36.567390 9894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:46:36.587388 9894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:46:36.591085 9894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:46:36.591282 9894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:46:36.591986 9894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:46:36.593106 9894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:46:36.593160 9894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:36.593209 9894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:46:36.593230 9894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:36.604027 9894 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:43205
I20260810 01:46:36.604079 10096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:43205 every 8 connection(s)
I20260810 01:46:36.605157 9894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:46:36.610587 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 9894
I20260810 01:46:36.610741 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance
I20260810 01:46:36.611112 9526 external_mini_cluster.cc:1428] Setting key bc1dd324b5f89b7dcb781eff92b31940
Aug 10 01:46:36 dist-test-slave-kvfs krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326396, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:36.624372 10101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:36.607678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:34523 (local address 127.9.77.190:42697)
0810 01:46:36.608109 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:36.608113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:36.609035 (+ 922us) server_negotiation.cc:407] Connection header received
0810 01:46:36.610625 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:36.610630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:36.610703 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:36.610843 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:36.612430 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:36.613203 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:36.614512 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:36.614918 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:36.615053 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:36.618486 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:36.618493 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:36.618499 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:36.618541 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:36.620918 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:36.621617 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:36.621624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:36.621628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:36.621776 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:36.622304 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:36.622310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:36.622313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:36.622667 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:36.622812 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:36.623652 (+ 840us) server_negotiation.cc:299] Negotiation successful
0810 01:46:36.623854 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":135,"threads_started":1}
I20260810 01:46:36.625363 10102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:36.608323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:42697 (local address 127.9.77.130:34523)
0810 01:46:36.608858 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:36.608908 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:36.610373 (+ 1465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:36.611056 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:36.611064 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:36.611546 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:46:36.612267 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:36.612288 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:36.613355 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:36.613358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:36.614376 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:36.614387 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:36.615083 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:36.615086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:36.615271 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:36.616112 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:46:36.616140 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:46:36.618264 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:46:36.621109 (+ 2845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:36.621116 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:36.621134 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:36.621477 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:36.621979 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:36.621989 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:36.621991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:36.622176 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:36.622818 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:46:36.622825 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:46:36.623321 (+ 496us) client_negotiation.cc:769] Sending connection context
0810 01:46:36.623582 (+ 261us) client_negotiation.cc:240] Negotiation successful
0810 01:46:36.623971 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":128,"threads_started":1}
WARNING: no policy specified for kudu/127.9.77.131@KRBTEST.COM; defaulting to no policy
I20260810 01:46:36.626951 10097 heartbeater.cc:347] Connected to a master server at 127.9.77.190:42697
I20260810 01:46:36.627249 10097 heartbeater.cc:464] Registering TS with master...
I20260810 01:46:36.627871 10097 heartbeater.cc:511] Master 127.9.77.190:42697 requested a full tablet report, sending...
I20260810 01:46:36.629207 9602 ts_manager.cc:194] Registered new tserver with Master: 419ed6fa8ae04406828c40f33be38569 (127.9.77.130:43205)
I20260810 01:46:36.629904 9602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.130@KRBTEST.COM'} at 127.9.77.130:34523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
W20260810 01:46:36.843801 9526 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.9.77.131: real 0.230s user 0.000s sys 0.006s
I20260810 01:46:36.844551 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:0
--local_ip_for_outbound_sockets=127.9.77.131
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:42697
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:40629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:46:36.979796 10107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:46:36.980154 10107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:46:36.980202 10107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:46:36.984228 10107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:46:36.984282 10107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:46:36.984339 10107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:46:36.989760 10107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:40629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:42697
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:46:36.991063 10107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:46:36.992307 10107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 01:46:37.004055 10107 server_base.cc:1034] running on GCE node
W20260810 01:46:37.003962 10114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:37.003979 10113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:37.003968 10116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:37.005028 10107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:46:37.005752 10107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:46:37.006937 10107 hybrid_clock.cc:648] HybridClock initialized: now 1786326397006912 us; error 37 us; skew 500 ppm
Aug 10 01:46:37 dist-test-slave-kvfs krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326397, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:46:37.010635 10107 init.cc:377] Logged in from keytab as kudu/127.9.77.131@KRBTEST.COM (short username kudu)
I20260810 01:46:37.012001 10107 webserver.cc:460] Webserver started at http://127.9.77.131:38961/ using document root <none> and password file <none>
I20260810 01:46:37.012835 10107 fs_manager.cc:359] Metadata directory not provided
I20260810 01:46:37.012944 10107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:46:37.013257 10107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:46:37.015893 10107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-2/data/instance:
uuid: "52dc00dd8899467fb9a0f26662d0f598"
format_stamp: "Formatted at 2026-08-10 01:46:37 on dist-test-slave-kvfs"
server_key: "8fbbd2b6ab63292a75b76ea9ceedccc7"
server_key_iv: "36e1c72cee6e8f39637c14584594fbf2"
server_key_version: "encryptionkey@0"
I20260810 01:46:37.016666 10107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance:
uuid: "52dc00dd8899467fb9a0f26662d0f598"
format_stamp: "Formatted at 2026-08-10 01:46:37 on dist-test-slave-kvfs"
server_key: "8fbbd2b6ab63292a75b76ea9ceedccc7"
server_key_iv: "36e1c72cee6e8f39637c14584594fbf2"
server_key_version: "encryptionkey@0"
I20260810 01:46:37.021463 10107 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 01:46:37.024520 10125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:37.026124 10107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:46:37.026291 10107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "52dc00dd8899467fb9a0f26662d0f598"
format_stamp: "Formatted at 2026-08-10 01:46:37 on dist-test-slave-kvfs"
server_key: "8fbbd2b6ab63292a75b76ea9ceedccc7"
server_key_iv: "36e1c72cee6e8f39637c14584594fbf2"
server_key_version: "encryptionkey@0"
I20260810 01:46:37.026425 10107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:46:37.038383 10107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:46:37.042251 10107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:46:37.042501 10107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:46:37.043254 10107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:46:37.044378 10107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:46:37.044454 10107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:37.044536 10107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:46:37.044592 10107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:37.055409 10107 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:39201
I20260810 01:46:37.055454 10299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:39201 every 8 connection(s)
I20260810 01:46:37.056586 10107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:46:37.064465 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 10107
I20260810 01:46:37.064631 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance
I20260810 01:46:37.065018 9526 external_mini_cluster.cc:1428] Setting key a591f89c814903005f9d4483e4c7e6ed
Aug 10 01:46:37 dist-test-slave-kvfs krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326397, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:37.073598 10101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:37.058881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:52625 (local address 127.9.77.190:42697)
0810 01:46:37.059072 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:37.059078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:37.060231 (+ 1153us) server_negotiation.cc:407] Connection header received
0810 01:46:37.061786 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:37.061790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:37.061848 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:37.061930 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:37.063364 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:37.064006 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:37.065864 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:37.066142 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:37.066298 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:37.069532 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:37.069536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:37.069541 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:37.069574 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:37.071134 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:37.071715 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:37.071719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:37.071721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:37.071840 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:37.072284 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:37.072287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:37.072289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:37.072512 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:37.072623 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:37.073201 (+ 578us) server_negotiation.cc:299] Negotiation successful
0810 01:46:37.073363 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 01:46:37.074618 10302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:37.059381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:42697 (local address 127.9.77.131:52625)
0810 01:46:37.059975 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:37.060104 (+ 129us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:37.061529 (+ 1425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:37.062130 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:37.062137 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:37.062560 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:46:37.063180 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:37.063195 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:37.064209 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:37.064214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:37.065706 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:37.065719 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:37.066342 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:37.066347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:37.066605 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:37.067507 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:46:37.067525 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:46:37.069356 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:46:37.071293 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:37.071300 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:37.071318 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:37.071603 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:37.071947 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:37.071950 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:37.071953 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:37.072187 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:37.072630 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:46:37.072640 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:46:37.072939 (+ 299us) client_negotiation.cc:769] Sending connection context
0810 01:46:37.073145 (+ 206us) client_negotiation.cc:240] Negotiation successful
0810 01:46:37.073530 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":141,"threads_started":1}
I20260810 01:46:37.075981 10300 heartbeater.cc:347] Connected to a master server at 127.9.77.190:42697
I20260810 01:46:37.076342 10300 heartbeater.cc:464] Registering TS with master...
I20260810 01:46:37.076941 10300 heartbeater.cc:511] Master 127.9.77.190:42697 requested a full tablet report, sending...
I20260810 01:46:37.078220 9602 ts_manager.cc:194] Registered new tserver with Master: 52dc00dd8899467fb9a0f26662d0f598 (127.9.77.131:39201)
I20260810 01:46:37.079170 9602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.131@KRBTEST.COM'} at 127.9.77.131:52625
I20260810 01:46:37.080662 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 01:46:37 dist-test-slave-kvfs krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326397, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 01:46:37 dist-test-slave-kvfs krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326397, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.77.129@KRBTEST.COM
I20260810 01:46:37.115666 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:37.104083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55958 (local address 127.9.77.129:33025)
0810 01:46:37.104486 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:37.104495 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:37.104530 (+ 35us) server_negotiation.cc:407] Connection header received
0810 01:46:37.104661 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:37.104670 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:37.104965 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:37.105172 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:37.106017 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:37.107010 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:37.107991 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:37.108529 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:37.108672 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:37.111620 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:37.111631 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:37.111651 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:37.111682 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:37.113531 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:37.114034 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:37.114041 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:37.114042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:37.114137 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:37.114449 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:37.114452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:37.114454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:37.114771 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:37.115069 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:37.115248 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 01:46:37.115442 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":119,"threads_started":1}
W20260810 01:46:37.117093 9757 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55958
Aug 10 01:46:37 dist-test-slave-kvfs krb5kdc[9538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 01:46:37 dist-test-slave-kvfs krb5kdc[9538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.129@KRBTEST.COM
I20260810 01:46:37.152365 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:37.141776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55960 (local address 127.9.77.129:33025)
0810 01:46:37.141946 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:37.141952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:37.142129 (+ 177us) server_negotiation.cc:407] Connection header received
0810 01:46:37.142275 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:37.142279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:37.142332 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:37.142437 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:37.143309 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:37.144248 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:37.145314 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:37.145677 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:37.145766 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:37.148773 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:37.148776 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:37.148778 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:37.148806 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:37.150422 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:37.150924 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:37.150929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:37.150936 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:37.151021 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:37.151317 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:37.151320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:37.151322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:37.151536 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:37.151695 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:37.151955 (+ 260us) server_negotiation.cc:299] Negotiation successful
0810 01:46:37.152164 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 01:46:37.153738 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 9697
I20260810 01:46:37.174207 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 9894
I20260810 01:46:37.208150 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 10107
I20260810 01:46:37.226328 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 9555
2026-08-10T01:46:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4351 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:46:37 dist-test-slave-kvfs krb5kdc[10317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:46:37 dist-test-slave-kvfs krb5kdc[10317](info): set up 2 sockets
Aug 10 01:46:37 dist-test-slave-kvfs krb5kdc[10317](info): commencing operation
krb5kdc: starting...
W20260810 01:46:38.553967 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.291s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:46:38 dist-test-slave-kvfs krb5kdc[10317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:46:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:46:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:46:38.708629 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:40381
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:43739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:40381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:46:38.857064 10338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:46:38.857546 10338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:46:38.857605 10338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:46:38.861909 10338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:46:38.861971 10338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:46:38.861990 10338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:46:38.862007 10338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:46:38.862023 10338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:46:38.867833 10338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:43739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:40381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:40381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:46:38.869397 10338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:46:38.870716 10338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:46:38.881217 10350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:38.881217 10348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:38.881337 10347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:38.881775 10338 server_base.cc:1034] running on GCE node
I20260810 01:46:38.882476 10338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:46:38.883816 10338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:46:38.885044 10338 hybrid_clock.cc:648] HybridClock initialized: now 1786326398885022 us; error 36 us; skew 500 ppm
Aug 10 01:46:38 dist-test-slave-kvfs krb5kdc[10317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326398, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:46:38.893111 10338 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:46:38.894390 10338 webserver.cc:460] Webserver started at http://127.9.77.190:44839/ using document root <none> and password file <none>
I20260810 01:46:38.895143 10338 fs_manager.cc:359] Metadata directory not provided
I20260810 01:46:38.895241 10338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:46:38.895527 10338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:46:38.898264 10338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "c40a0e946c244aa881c782906d7bbe66"
format_stamp: "Formatted at 2026-08-10 01:46:38 on dist-test-slave-kvfs"
server_key: "36779f66df15569b77110b9b4dccc164"
server_key_iv: "7cb48df86201e88a3d9e5b75fbd900e4"
server_key_version: "encryptionkey@0"
I20260810 01:46:38.898932 10338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "c40a0e946c244aa881c782906d7bbe66"
format_stamp: "Formatted at 2026-08-10 01:46:38 on dist-test-slave-kvfs"
server_key: "36779f66df15569b77110b9b4dccc164"
server_key_iv: "7cb48df86201e88a3d9e5b75fbd900e4"
server_key_version: "encryptionkey@0"
I20260810 01:46:38.903736 10338 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260810 01:46:38.906864 10360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:38.908648 10338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:46:38.908835 10338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "c40a0e946c244aa881c782906d7bbe66"
format_stamp: "Formatted at 2026-08-10 01:46:38 on dist-test-slave-kvfs"
server_key: "36779f66df15569b77110b9b4dccc164"
server_key_iv: "7cb48df86201e88a3d9e5b75fbd900e4"
server_key_version: "encryptionkey@0"
I20260810 01:46:38.908977 10338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:46:38.923264 10338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:46:38.927500 10338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:46:38.927781 10338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:46:38.936766 10338 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:40381
I20260810 01:46:38.936781 10445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:40381 every 8 connection(s)
I20260810 01:46:38.937983 10338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:46:38.939158 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 10338
I20260810 01:46:38.939301 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:46:38.939644 9526 external_mini_cluster.cc:1428] Setting key 1c5db54cf53f7cb15d3b21b167e6eb4e
I20260810 01:46:38.942358 10448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:38.950336 10448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c40a0e946c244aa881c782906d7bbe66: Bootstrap starting.
Aug 10 01:46:38 dist-test-slave-kvfs krb5kdc[10317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:38.953912 10448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c40a0e946c244aa881c782906d7bbe66: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:38.955003 10448 log.cc:824] T 00000000000000000000000000000000 P c40a0e946c244aa881c782906d7bbe66: Log is configured to *not* fsync() on all Append() calls
I20260810 01:46:38.957849 10452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:38.941460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60056 (local address 127.9.77.190:40381)
0810 01:46:38.942032 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:38.942055 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:38.942109 (+ 54us) server_negotiation.cc:407] Connection header received
0810 01:46:38.943092 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:38.943120 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:38.943601 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:38.944014 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:38.944859 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:38.946561 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:38.947540 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:38.948316 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:38.948504 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:38.951481 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:38.951488 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:38.951506 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:38.951543 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:38.953840 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:38.954368 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:38.954375 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:38.954382 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:38.954535 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:38.954918 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:38.954921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:38.954923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:38.955576 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:38.955769 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:38.956341 (+ 572us) server_negotiation.cc:299] Negotiation successful
0810 01:46:38.956702 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":169,"threads_started":1}
I20260810 01:46:38.958027 10448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c40a0e946c244aa881c782906d7bbe66: No bootstrap required, opened a new log
I20260810 01:46:38.962208 10448 raft_consensus.cc:348] T 00000000000000000000000000000000 P c40a0e946c244aa881c782906d7bbe66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40a0e946c244aa881c782906d7bbe66" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 40381 } }
I20260810 01:46:38.962568 10448 raft_consensus.cc:374] T 00000000000000000000000000000000 P c40a0e946c244aa881c782906d7bbe66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:38.962666 10448 raft_consensus.cc:729] T 00000000000000000000000000000000 P c40a0e946c244aa881c782906d7bbe66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c40a0e946c244aa881c782906d7bbe66, State: Initialized, Role: FOLLOWER
I20260810 01:46:38.963215 10448 consensus_queue.cc:260] T 00000000000000000000000000000000 P c40a0e946c244aa881c782906d7bbe66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40a0e946c244aa881c782906d7bbe66" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 40381 } }
I20260810 01:46:38.963369 10448 raft_consensus.cc:388] T 00000000000000000000000000000000 P c40a0e946c244aa881c782906d7bbe66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:46:38.963460 10448 raft_consensus.cc:482] T 00000000000000000000000000000000 P c40a0e946c244aa881c782906d7bbe66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:46:38.963567 10448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c40a0e946c244aa881c782906d7bbe66 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:38.964747 10448 raft_consensus.cc:504] T 00000000000000000000000000000000 P c40a0e946c244aa881c782906d7bbe66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40a0e946c244aa881c782906d7bbe66" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 40381 } }
I20260810 01:46:38.965175 10448 leader_election.cc:302] T 00000000000000000000000000000000 P c40a0e946c244aa881c782906d7bbe66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c40a0e946c244aa881c782906d7bbe66; no voters:
I20260810 01:46:38.965584 10448 leader_election.cc:288] T 00000000000000000000000000000000 P c40a0e946c244aa881c782906d7bbe66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:46:38.965739 10455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c40a0e946c244aa881c782906d7bbe66 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:46:38.966042 10455 raft_consensus.cc:686] T 00000000000000000000000000000000 P c40a0e946c244aa881c782906d7bbe66 [term 1 LEADER]: Becoming Leader. State: Replica: c40a0e946c244aa881c782906d7bbe66, State: Running, Role: LEADER
I20260810 01:46:38.966413 10455 consensus_queue.cc:237] T 00000000000000000000000000000000 P c40a0e946c244aa881c782906d7bbe66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40a0e946c244aa881c782906d7bbe66" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 40381 } }
I20260810 01:46:38.967226 10448 sys_catalog.cc:564] T 00000000000000000000000000000000 P c40a0e946c244aa881c782906d7bbe66 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:46:38.968645 10457 sys_catalog.cc:455] T 00000000000000000000000000000000 P c40a0e946c244aa881c782906d7bbe66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c40a0e946c244aa881c782906d7bbe66. Latest consensus state: current_term: 1 leader_uuid: "c40a0e946c244aa881c782906d7bbe66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40a0e946c244aa881c782906d7bbe66" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 40381 } } }
I20260810 01:46:38.968806 10457 sys_catalog.cc:458] T 00000000000000000000000000000000 P c40a0e946c244aa881c782906d7bbe66 [sys.catalog]: This master's current role is: LEADER
I20260810 01:46:38.969190 10456 sys_catalog.cc:455] T 00000000000000000000000000000000 P c40a0e946c244aa881c782906d7bbe66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c40a0e946c244aa881c782906d7bbe66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40a0e946c244aa881c782906d7bbe66" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 40381 } } }
I20260810 01:46:38.969295 10456 sys_catalog.cc:458] T 00000000000000000000000000000000 P c40a0e946c244aa881c782906d7bbe66 [sys.catalog]: This master's current role is: LEADER
I20260810 01:46:38.969422 10471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:46:38.973052 10471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:46:38.979208 10471 catalog_manager.cc:1347] Generated new cluster ID: 2e1f861ec14249f28c1aa64cda75db15
I20260810 01:46:38.979301 10471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:46:39.002921 10471 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:46:39.004217 10471 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:46:39.013876 10471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c40a0e946c244aa881c782906d7bbe66: Generated new TSK 0
I20260810 01:46:39.014937 10471 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.77.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
I20260810 01:46:39.055430 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:0
--local_ip_for_outbound_sockets=127.9.77.129
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:40381
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:43739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:46:39.196723 10494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:46:39.197070 10494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:46:39.197120 10494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:46:39.201228 10494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:46:39.201304 10494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:46:39.201365 10494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:46:39.207204 10494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:43739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:40381
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:46:39.208766 10494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:46:39.210115 10494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:46:39.222846 10507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:39.222868 10501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:39.223316 10494 server_base.cc:1034] running on GCE node
W20260810 01:46:39.222868 10504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:39.223939 10494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:46:39.224903 10494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:46:39.226104 10494 hybrid_clock.cc:648] HybridClock initialized: now 1786326399226063 us; error 57 us; skew 500 ppm
Aug 10 01:46:39 dist-test-slave-kvfs krb5kdc[10317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326399, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:46:39.229719 10494 init.cc:377] Logged in from keytab as kudu/127.9.77.129@KRBTEST.COM (short username kudu)
I20260810 01:46:39.230937 10494 webserver.cc:460] Webserver started at http://127.9.77.129:38305/ using document root <none> and password file <none>
I20260810 01:46:39.231683 10494 fs_manager.cc:359] Metadata directory not provided
I20260810 01:46:39.231787 10494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:46:39.232144 10494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:46:39.234773 10494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-0/data/instance:
uuid: "fe139edec2384aca8046d2de9c18fb99"
format_stamp: "Formatted at 2026-08-10 01:46:39 on dist-test-slave-kvfs"
server_key: "e0346f052318c448107f2d10676ff039"
server_key_iv: "7ee392d505dc88c05bf4036cf22d6971"
server_key_version: "encryptionkey@0"
I20260810 01:46:39.235508 10494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance:
uuid: "fe139edec2384aca8046d2de9c18fb99"
format_stamp: "Formatted at 2026-08-10 01:46:39 on dist-test-slave-kvfs"
server_key: "e0346f052318c448107f2d10676ff039"
server_key_iv: "7ee392d505dc88c05bf4036cf22d6971"
server_key_version: "encryptionkey@0"
I20260810 01:46:39.240342 10494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260810 01:46:39.244439 10516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:39.246567 10494 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 01:46:39.246717 10494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "fe139edec2384aca8046d2de9c18fb99"
format_stamp: "Formatted at 2026-08-10 01:46:39 on dist-test-slave-kvfs"
server_key: "e0346f052318c448107f2d10676ff039"
server_key_iv: "7ee392d505dc88c05bf4036cf22d6971"
server_key_version: "encryptionkey@0"
I20260810 01:46:39.246877 10494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:46:39.270382 10494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:46:39.274731 10494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:46:39.275005 10494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:46:39.275859 10494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:46:39.277037 10494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:46:39.277102 10494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:39.277177 10494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:46:39.277211 10494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:39.288396 10494 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:40657
I20260810 01:46:39.288443 10685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:40657 every 8 connection(s)
I20260810 01:46:39.289666 10494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:46:39.296164 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 10494
I20260810 01:46:39.296289 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance
I20260810 01:46:39.296619 9526 external_mini_cluster.cc:1428] Setting key ca1e452f0932ee623a55073a4d45da13
Aug 10 01:46:39 dist-test-slave-kvfs krb5kdc[10317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326399, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:39.306187 10452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:39.292144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:52693 (local address 127.9.77.190:40381)
0810 01:46:39.292331 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:39.292336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:39.293523 (+ 1187us) server_negotiation.cc:407] Connection header received
0810 01:46:39.294681 (+ 1158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:39.294686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:39.294761 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:39.294895 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:39.296486 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:39.297156 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:39.298665 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:39.298923 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:39.299036 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:39.302011 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:39.302015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:39.302019 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:39.302052 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:39.303713 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:39.304388 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:39.304392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:39.304393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:39.304477 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:39.304846 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:39.304849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:39.304851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:39.305078 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:39.305193 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:39.305806 (+ 613us) server_negotiation.cc:299] Negotiation successful
0810 01:46:39.305973 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 01:46:39.307288 10689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:39.292643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:40381 (local address 127.9.77.129:52693)
0810 01:46:39.293323 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:39.293380 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:39.294425 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:39.295107 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:39.295115 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:39.295571 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:46:39.296305 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:39.296328 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:39.297290 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:39.297294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:39.298538 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:39.298547 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:39.299085 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:39.299089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:39.299296 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:39.300013 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:46:39.300082 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:46:39.301824 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:46:39.303855 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:39.303862 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:39.303883 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:39.304262 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:39.304572 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:39.304577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:39.304579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:39.304742 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:39.305189 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:46:39.305197 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:46:39.305524 (+ 327us) client_negotiation.cc:769] Sending connection context
0810 01:46:39.305741 (+ 217us) client_negotiation.cc:240] Negotiation successful
0810 01:46:39.306152 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":243,"threads_started":1}
I20260810 01:46:39.309129 10686 heartbeater.cc:347] Connected to a master server at 127.9.77.190:40381
WARNING: no policy specified for kudu/127.9.77.130@KRBTEST.COM; defaulting to no policy
I20260810 01:46:39.309561 10686 heartbeater.cc:464] Registering TS with master...
I20260810 01:46:39.310300 10686 heartbeater.cc:511] Master 127.9.77.190:40381 requested a full tablet report, sending...
I20260810 01:46:39.312952 10382 ts_manager.cc:194] Registered new tserver with Master: fe139edec2384aca8046d2de9c18fb99 (127.9.77.129:40657)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.130@KRBTEST.COM" created.
I20260810 01:46:39.314677 10382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.129@KRBTEST.COM'} at 127.9.77.129:52693
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
I20260810 01:46:39.335788 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:0
--local_ip_for_outbound_sockets=127.9.77.130
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:40381
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:43739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:46:39.474591 10694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:46:39.474936 10694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:46:39.474987 10694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:46:39.479236 10694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:46:39.479306 10694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:46:39.479395 10694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:46:39.485409 10694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:43739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:40381
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:46:39.486901 10694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:46:39.488284 10694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:46:39.500332 10700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:39.500442 10704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:39.500406 10694 server_base.cc:1034] running on GCE node
W20260810 01:46:39.500355 10701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:39.501199 10694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:46:39.502082 10694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:46:39.503297 10694 hybrid_clock.cc:648] HybridClock initialized: now 1786326399503269 us; error 38 us; skew 500 ppm
Aug 10 01:46:39 dist-test-slave-kvfs krb5kdc[10317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326399, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:46:39.506837 10694 init.cc:377] Logged in from keytab as kudu/127.9.77.130@KRBTEST.COM (short username kudu)
I20260810 01:46:39.507975 10694 webserver.cc:460] Webserver started at http://127.9.77.130:45359/ using document root <none> and password file <none>
I20260810 01:46:39.508760 10694 fs_manager.cc:359] Metadata directory not provided
I20260810 01:46:39.508868 10694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:46:39.509176 10694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:46:39.511832 10694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-1/data/instance:
uuid: "cf3fa56517be4eb8abb73705f5d3a6b6"
format_stamp: "Formatted at 2026-08-10 01:46:39 on dist-test-slave-kvfs"
server_key: "5f1efda7c9cd722f9d03393809cb0293"
server_key_iv: "571dbee8e59392413116dd8ce104e1a9"
server_key_version: "encryptionkey@0"
I20260810 01:46:39.512624 10694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance:
uuid: "cf3fa56517be4eb8abb73705f5d3a6b6"
format_stamp: "Formatted at 2026-08-10 01:46:39 on dist-test-slave-kvfs"
server_key: "5f1efda7c9cd722f9d03393809cb0293"
server_key_iv: "571dbee8e59392413116dd8ce104e1a9"
server_key_version: "encryptionkey@0"
I20260810 01:46:39.517874 10694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 01:46:39.521034 10714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:39.522846 10694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 01:46:39.523028 10694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "cf3fa56517be4eb8abb73705f5d3a6b6"
format_stamp: "Formatted at 2026-08-10 01:46:39 on dist-test-slave-kvfs"
server_key: "5f1efda7c9cd722f9d03393809cb0293"
server_key_iv: "571dbee8e59392413116dd8ce104e1a9"
server_key_version: "encryptionkey@0"
I20260810 01:46:39.523182 10694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:46:39.539593 10694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:46:39.543953 10694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:46:39.544224 10694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:46:39.545008 10694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:46:39.546154 10694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:46:39.546252 10694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:39.546341 10694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:46:39.546399 10694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:39.558068 10694 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:44619
I20260810 01:46:39.558110 10880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:44619 every 8 connection(s)
I20260810 01:46:39.559357 10694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:46:39.566546 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 10694
I20260810 01:46:39.566689 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance
I20260810 01:46:39.567050 9526 external_mini_cluster.cc:1428] Setting key 7534d78de3e75805b729131223e128b9
Aug 10 01:46:39 dist-test-slave-kvfs krb5kdc[10317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326399, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:39.576741 10452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:39.561866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:59577 (local address 127.9.77.190:40381)
0810 01:46:39.562074 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:39.562080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:39.563180 (+ 1100us) server_negotiation.cc:407] Connection header received
0810 01:46:39.564405 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:39.564409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:39.564470 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:39.564559 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:39.566026 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:39.566940 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:39.568811 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:39.569197 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:39.569314 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:39.572511 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:39.572515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:39.572519 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:39.572558 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:39.574176 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:39.574834 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:39.574838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:39.574840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:39.574935 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:39.575315 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:39.575318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:39.575320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:39.575594 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:39.575702 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:39.576379 (+ 677us) server_negotiation.cc:299] Negotiation successful
0810 01:46:39.576543 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 01:46:39.577862 10884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:39.562396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:40381 (local address 127.9.77.130:59577)
0810 01:46:39.562988 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:39.563041 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:39.564152 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:39.564769 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:39.564776 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:39.565233 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:46:39.565847 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:39.565863 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:39.567087 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:39.567090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:39.568672 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:39.568682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:39.569392 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:39.569395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:39.569564 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:39.570316 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:46:39.570334 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:46:39.572317 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:46:39.574329 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:39.574336 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:39.574355 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:39.574697 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:39.575031 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:39.575036 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:39.575038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:39.575199 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:39.575696 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:46:39.575705 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:46:39.576062 (+ 357us) client_negotiation.cc:769] Sending connection context
0810 01:46:39.576322 (+ 260us) client_negotiation.cc:240] Negotiation successful
0810 01:46:39.576751 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":132,"threads_started":1}
I20260810 01:46:39.579607 10881 heartbeater.cc:347] Connected to a master server at 127.9.77.190:40381
I20260810 01:46:39.580006 10881 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.9.77.131@KRBTEST.COM; defaulting to no policy
I20260810 01:46:39.580830 10881 heartbeater.cc:511] Master 127.9.77.190:40381 requested a full tablet report, sending...
I20260810 01:46:39.582381 10382 ts_manager.cc:194] Registered new tserver with Master: cf3fa56517be4eb8abb73705f5d3a6b6 (127.9.77.130:44619)
I20260810 01:46:39.583238 10382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.130@KRBTEST.COM'} at 127.9.77.130:59577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
I20260810 01:46:39.612334 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:0
--local_ip_for_outbound_sockets=127.9.77.131
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:40381
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:43739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:46:39.747486 10889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:46:39.747843 10889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:46:39.747892 10889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:46:39.752226 10889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:46:39.752310 10889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:46:39.752403 10889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:46:39.758275 10889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:43739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:40381
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:46:39.759733 10889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:46:39.761135 10889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:46:39.773188 10899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:39.773221 10896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:39.773434 10889 server_base.cc:1034] running on GCE node
W20260810 01:46:39.773197 10906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:39.774121 10889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:46:39.774893 10889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:46:39.776131 10889 hybrid_clock.cc:648] HybridClock initialized: now 1786326399776114 us; error 27 us; skew 500 ppm
Aug 10 01:46:39 dist-test-slave-kvfs krb5kdc[10317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326399, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:46:39.779448 10889 init.cc:377] Logged in from keytab as kudu/127.9.77.131@KRBTEST.COM (short username kudu)
I20260810 01:46:39.780571 10889 webserver.cc:460] Webserver started at http://127.9.77.131:39373/ using document root <none> and password file <none>
I20260810 01:46:39.781275 10889 fs_manager.cc:359] Metadata directory not provided
I20260810 01:46:39.781374 10889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:46:39.781657 10889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:46:39.784196 10889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-2/data/instance:
uuid: "fc469ccaa843461e8421ecfe8b44b210"
format_stamp: "Formatted at 2026-08-10 01:46:39 on dist-test-slave-kvfs"
server_key: "c2b97a80ca5487f33e8689ccabe356c3"
server_key_iv: "57f7769f2fdd346e8dbbf5f6d7a73e9d"
server_key_version: "encryptionkey@0"
I20260810 01:46:39.784878 10889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance:
uuid: "fc469ccaa843461e8421ecfe8b44b210"
format_stamp: "Formatted at 2026-08-10 01:46:39 on dist-test-slave-kvfs"
server_key: "c2b97a80ca5487f33e8689ccabe356c3"
server_key_iv: "57f7769f2fdd346e8dbbf5f6d7a73e9d"
server_key_version: "encryptionkey@0"
I20260810 01:46:39.789561 10889 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 01:46:39.792598 10914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:39.794174 10889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 01:46:39.794351 10889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "fc469ccaa843461e8421ecfe8b44b210"
format_stamp: "Formatted at 2026-08-10 01:46:39 on dist-test-slave-kvfs"
server_key: "c2b97a80ca5487f33e8689ccabe356c3"
server_key_iv: "57f7769f2fdd346e8dbbf5f6d7a73e9d"
server_key_version: "encryptionkey@0"
I20260810 01:46:39.794486 10889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:46:39.809798 10889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:46:39.813905 10889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:46:39.814162 10889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:46:39.814889 10889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:46:39.815899 10889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:46:39.815973 10889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:39.816080 10889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:46:39.816126 10889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:39.827199 10889 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:35295
I20260810 01:46:39.827286 11083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:35295 every 8 connection(s)
I20260810 01:46:39.828536 10889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:46:39.833295 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 10889
I20260810 01:46:39.833447 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance
I20260810 01:46:39.833854 9526 external_mini_cluster.cc:1428] Setting key e89350aae07eadd914aca3e681c97ce9
Aug 10 01:46:39 dist-test-slave-kvfs krb5kdc[10317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326399, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:39.845749 10452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:39.831035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:52595 (local address 127.9.77.190:40381)
0810 01:46:39.831216 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:39.831220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:39.832619 (+ 1399us) server_negotiation.cc:407] Connection header received
0810 01:46:39.834031 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:39.834038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:39.834115 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:39.834198 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:39.836076 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:39.836881 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:39.838240 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:39.838490 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:39.838587 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:39.841705 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:39.841709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:39.841711 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:39.841769 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:39.843387 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:39.844123 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:39.844126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:39.844128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:39.844222 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:39.844559 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:39.844565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:39.844567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:39.844788 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:39.844872 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:39.845439 (+ 567us) server_negotiation.cc:299] Negotiation successful
0810 01:46:39.845566 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 01:46:39.846866 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:39.831502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:40381 (local address 127.9.77.131:52595)
0810 01:46:39.832425 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:39.832477 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:39.833741 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:39.834447 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:39.834455 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:39.835085 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:46:39.835893 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:39.835912 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:39.837028 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:39.837031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:39.838096 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:39.838107 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:39.838669 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:39.838673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:39.838894 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:39.839667 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:46:39.839686 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:46:39.841506 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:46:39.843502 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:39.843508 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:39.843523 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:39.843971 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:39.844313 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:39.844316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:39.844317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:39.844468 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:39.844883 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:46:39.844890 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:46:39.845182 (+ 292us) client_negotiation.cc:769] Sending connection context
0810 01:46:39.845358 (+ 176us) client_negotiation.cc:240] Negotiation successful
0810 01:46:39.845688 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":126,"threads_started":1}
I20260810 01:46:39.848515 11087 heartbeater.cc:347] Connected to a master server at 127.9.77.190:40381
I20260810 01:46:39.848891 11087 heartbeater.cc:464] Registering TS with master...
I20260810 01:46:39.849524 11087 heartbeater.cc:511] Master 127.9.77.190:40381 requested a full tablet report, sending...
I20260810 01:46:39.850801 10382 ts_manager.cc:194] Registered new tserver with Master: fc469ccaa843461e8421ecfe8b44b210 (127.9.77.131:35295)
I20260810 01:46:39.851478 10382 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.131@KRBTEST.COM'} at 127.9.77.131:52595
I20260810 01:46:39.860911 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 01:46:39.874258 10452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:39.865613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60064 (local address 127.9.77.190:40381)
0810 01:46:39.865770 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:39.865779 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:39.865965 (+ 186us) server_negotiation.cc:407] Connection header received
0810 01:46:39.866119 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:39.866122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:39.866178 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:39.866264 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:39.867128 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:39.867785 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:39.868898 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:39.869193 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:39.869289 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:39.870865 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:39.870868 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:39.870871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:39.870934 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:39.872587 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:39.873009 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:39.873011 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:39.873013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:39.873096 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:39.873334 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:39.873336 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:39.873337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:39.873605 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:39.873721 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:39.873983 (+ 262us) server_negotiation.cc:299] Negotiation successful
0810 01:46:39.874107 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 01:46:39.880638 10382 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 01:46:39.884668 10382 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:46:39.902470 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:39.895677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32888 (local address 127.9.77.129:40657)
0810 01:46:39.896811 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:39.896822 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:39.896860 (+ 38us) server_negotiation.cc:407] Connection header received
0810 01:46:39.897088 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:39.897096 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:39.897364 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:39.897519 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:46:39.898450 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:39.899690 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:39.900937 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:39.901679 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:39.901794 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:39.901975 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:39.902163 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 01:46:39.902282 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":995,"thread_start_us":166,"threads_started":1}
I20260810 01:46:39.902853 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:39.895410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:40657 (local address 127.0.0.1:32888)
0810 01:46:39.896756 (+ 1346us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:39.896769 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:39.896884 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:39.897752 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:39.897761 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:39.897819 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:46:39.898258 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:39.898264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:39.899825 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:39.899828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:39.900799 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:39.900809 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:39.901822 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:39.901826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:39.901935 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:39.901970 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 01:46:39.902119 (+ 149us) client_negotiation.cc:240] Negotiation successful
0810 01:46:39.902262 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1274,"mutex_wait_us":566,"spinlock_wait_cycles":23808,"thread_start_us":153,"threads_started":1}
I20260810 01:46:39.903494 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:39.895410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:35295 (local address 127.0.0.1:38886)
0810 01:46:39.895887 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:39.895950 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:39.896144 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:39.897750 (+ 1606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:39.897757 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:39.897802 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:46:39.898060 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:39.898067 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:39.899644 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:39.899647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:39.901254 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:39.901266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:39.903018 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:39.903022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:39.903108 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:39.903134 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 01:46:39.903186 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 01:46:39.903286 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":180,"spinlock_wait_cycles":3456,"thread_start_us":120,"threads_started":1}
I20260810 01:46:39.903509 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:39.896336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:44619 (local address 127.0.0.1:53450)
0810 01:46:39.897012 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:39.897027 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:39.897124 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:39.897897 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:39.897899 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:39.897911 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:46:39.898043 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:39.898049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:39.899460 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:39.899463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:39.901260 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:39.901266 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:39.902528 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:39.902531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:39.903137 (+ 606us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:39.903157 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 01:46:39.903222 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 01:46:39.903297 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":115,"thread_start_us":104,"threads_started":1}
I20260810 01:46:39.903996 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:39.896825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53450 (local address 127.9.77.130:44619)
0810 01:46:39.897329 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:39.897342 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:39.897367 (+ 25us) server_negotiation.cc:407] Connection header received
0810 01:46:39.897454 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:39.897461 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:39.897626 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:39.897771 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:46:39.898183 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:39.899307 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:39.901391 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:39.902216 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:39.902324 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:39.902465 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:39.903374 (+ 909us) server_negotiation.cc:299] Negotiation successful
0810 01:46:39.903483 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":137,"threads_started":1}
I20260810 01:46:39.904387 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:39.896231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38886 (local address 127.9.77.131:35295)
0810 01:46:39.897332 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:39.897343 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:39.897364 (+ 21us) server_negotiation.cc:407] Connection header received
0810 01:46:39.897437 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:39.897441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:39.897584 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:39.897794 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:46:39.898181 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:39.899495 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:39.902050 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:39.902865 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:39.903845 (+ 980us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:39.904009 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:39.904102 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 01:46:39.904197 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":978,"thread_start_us":223,"threads_started":1}
I20260810 01:46:39.907007 10777 tablet_service.cc:1467] Processing CreateTablet for tablet f5864b2535544a01bf40662c5ba95f43 (DEFAULT_TABLE table=test-table [id=8067273cff3b48c99652135f121a9e3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:46:39.907011 10587 tablet_service.cc:1467] Processing CreateTablet for tablet f5864b2535544a01bf40662c5ba95f43 (DEFAULT_TABLE table=test-table [id=8067273cff3b48c99652135f121a9e3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:46:39.907011 10990 tablet_service.cc:1467] Processing CreateTablet for tablet f5864b2535544a01bf40662c5ba95f43 (DEFAULT_TABLE table=test-table [id=8067273cff3b48c99652135f121a9e3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:46:39.908408 10587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5864b2535544a01bf40662c5ba95f43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:39.908408 10990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5864b2535544a01bf40662c5ba95f43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:39.908420 10777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5864b2535544a01bf40662c5ba95f43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:39.916308 11108 tablet_bootstrap.cc:492] T f5864b2535544a01bf40662c5ba95f43 P cf3fa56517be4eb8abb73705f5d3a6b6: Bootstrap starting.
I20260810 01:46:39.917712 11117 tablet_bootstrap.cc:492] T f5864b2535544a01bf40662c5ba95f43 P fc469ccaa843461e8421ecfe8b44b210: Bootstrap starting.
I20260810 01:46:39.919736 11108 tablet_bootstrap.cc:654] T f5864b2535544a01bf40662c5ba95f43 P cf3fa56517be4eb8abb73705f5d3a6b6: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:39.920184 11117 tablet_bootstrap.cc:654] T f5864b2535544a01bf40662c5ba95f43 P fc469ccaa843461e8421ecfe8b44b210: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:39.920830 11108 log.cc:824] T f5864b2535544a01bf40662c5ba95f43 P cf3fa56517be4eb8abb73705f5d3a6b6: Log is configured to *not* fsync() on all Append() calls
I20260810 01:46:39.920940 11117 log.cc:824] T f5864b2535544a01bf40662c5ba95f43 P fc469ccaa843461e8421ecfe8b44b210: Log is configured to *not* fsync() on all Append() calls
I20260810 01:46:39.921726 11111 tablet_bootstrap.cc:492] T f5864b2535544a01bf40662c5ba95f43 P fe139edec2384aca8046d2de9c18fb99: Bootstrap starting.
I20260810 01:46:39.923111 11108 tablet_bootstrap.cc:492] T f5864b2535544a01bf40662c5ba95f43 P cf3fa56517be4eb8abb73705f5d3a6b6: No bootstrap required, opened a new log
I20260810 01:46:39.923357 11108 ts_tablet_manager.cc:1397] T f5864b2535544a01bf40662c5ba95f43 P cf3fa56517be4eb8abb73705f5d3a6b6: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260810 01:46:39.923982 11111 tablet_bootstrap.cc:654] T f5864b2535544a01bf40662c5ba95f43 P fe139edec2384aca8046d2de9c18fb99: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:39.924947 11111 log.cc:824] T f5864b2535544a01bf40662c5ba95f43 P fe139edec2384aca8046d2de9c18fb99: Log is configured to *not* fsync() on all Append() calls
I20260810 01:46:39.925714 11117 tablet_bootstrap.cc:492] T f5864b2535544a01bf40662c5ba95f43 P fc469ccaa843461e8421ecfe8b44b210: No bootstrap required, opened a new log
I20260810 01:46:39.925976 11117 ts_tablet_manager.cc:1397] T f5864b2535544a01bf40662c5ba95f43 P fc469ccaa843461e8421ecfe8b44b210: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260810 01:46:39.927275 11111 tablet_bootstrap.cc:492] T f5864b2535544a01bf40662c5ba95f43 P fe139edec2384aca8046d2de9c18fb99: No bootstrap required, opened a new log
I20260810 01:46:39.927487 11111 ts_tablet_manager.cc:1397] T f5864b2535544a01bf40662c5ba95f43 P fe139edec2384aca8046d2de9c18fb99: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 01:46:39.927623 11108 raft_consensus.cc:348] T f5864b2535544a01bf40662c5ba95f43 P cf3fa56517be4eb8abb73705f5d3a6b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe139edec2384aca8046d2de9c18fb99" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 40657 } } peers { permanent_uuid: "fc469ccaa843461e8421ecfe8b44b210" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35295 } } peers { permanent_uuid: "cf3fa56517be4eb8abb73705f5d3a6b6" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44619 } }
I20260810 01:46:39.927947 11108 raft_consensus.cc:374] T f5864b2535544a01bf40662c5ba95f43 P cf3fa56517be4eb8abb73705f5d3a6b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:39.928007 11108 raft_consensus.cc:729] T f5864b2535544a01bf40662c5ba95f43 P cf3fa56517be4eb8abb73705f5d3a6b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf3fa56517be4eb8abb73705f5d3a6b6, State: Initialized, Role: FOLLOWER
I20260810 01:46:39.928601 11108 consensus_queue.cc:260] T f5864b2535544a01bf40662c5ba95f43 P cf3fa56517be4eb8abb73705f5d3a6b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe139edec2384aca8046d2de9c18fb99" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 40657 } } peers { permanent_uuid: "fc469ccaa843461e8421ecfe8b44b210" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35295 } } peers { permanent_uuid: "cf3fa56517be4eb8abb73705f5d3a6b6" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44619 } }
I20260810 01:46:39.929407 10881 heartbeater.cc:503] Master 127.9.77.190:40381 was elected leader, sending a full tablet report...
I20260810 01:46:39.930514 11117 raft_consensus.cc:348] T f5864b2535544a01bf40662c5ba95f43 P fc469ccaa843461e8421ecfe8b44b210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe139edec2384aca8046d2de9c18fb99" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 40657 } } peers { permanent_uuid: "fc469ccaa843461e8421ecfe8b44b210" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35295 } } peers { permanent_uuid: "cf3fa56517be4eb8abb73705f5d3a6b6" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44619 } }
I20260810 01:46:39.930812 11117 raft_consensus.cc:374] T f5864b2535544a01bf40662c5ba95f43 P fc469ccaa843461e8421ecfe8b44b210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:39.930863 11117 raft_consensus.cc:729] T f5864b2535544a01bf40662c5ba95f43 P fc469ccaa843461e8421ecfe8b44b210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc469ccaa843461e8421ecfe8b44b210, State: Initialized, Role: FOLLOWER
I20260810 01:46:39.931061 11108 ts_tablet_manager.cc:1428] T f5864b2535544a01bf40662c5ba95f43 P cf3fa56517be4eb8abb73705f5d3a6b6: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20260810 01:46:39.931414 11117 consensus_queue.cc:260] T f5864b2535544a01bf40662c5ba95f43 P fc469ccaa843461e8421ecfe8b44b210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe139edec2384aca8046d2de9c18fb99" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 40657 } } peers { permanent_uuid: "fc469ccaa843461e8421ecfe8b44b210" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35295 } } peers { permanent_uuid: "cf3fa56517be4eb8abb73705f5d3a6b6" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44619 } }
I20260810 01:46:39.932253 11087 heartbeater.cc:503] Master 127.9.77.190:40381 was elected leader, sending a full tablet report...
I20260810 01:46:39.932592 11117 ts_tablet_manager.cc:1428] T f5864b2535544a01bf40662c5ba95f43 P fc469ccaa843461e8421ecfe8b44b210: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260810 01:46:39.932968 11111 raft_consensus.cc:348] T f5864b2535544a01bf40662c5ba95f43 P fe139edec2384aca8046d2de9c18fb99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe139edec2384aca8046d2de9c18fb99" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 40657 } } peers { permanent_uuid: "fc469ccaa843461e8421ecfe8b44b210" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35295 } } peers { permanent_uuid: "cf3fa56517be4eb8abb73705f5d3a6b6" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44619 } }
I20260810 01:46:39.933223 11111 raft_consensus.cc:374] T f5864b2535544a01bf40662c5ba95f43 P fe139edec2384aca8046d2de9c18fb99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:39.933286 11111 raft_consensus.cc:729] T f5864b2535544a01bf40662c5ba95f43 P fe139edec2384aca8046d2de9c18fb99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe139edec2384aca8046d2de9c18fb99, State: Initialized, Role: FOLLOWER
I20260810 01:46:39.933808 11111 consensus_queue.cc:260] T f5864b2535544a01bf40662c5ba95f43 P fe139edec2384aca8046d2de9c18fb99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe139edec2384aca8046d2de9c18fb99" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 40657 } } peers { permanent_uuid: "fc469ccaa843461e8421ecfe8b44b210" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35295 } } peers { permanent_uuid: "cf3fa56517be4eb8abb73705f5d3a6b6" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44619 } }
I20260810 01:46:39.934939 11111 ts_tablet_manager.cc:1428] T f5864b2535544a01bf40662c5ba95f43 P fe139edec2384aca8046d2de9c18fb99: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20260810 01:46:39.935354 10686 heartbeater.cc:503] Master 127.9.77.190:40381 was elected leader, sending a full tablet report...
I20260810 01:46:40.017141 11127 raft_consensus.cc:482] T f5864b2535544a01bf40662c5ba95f43 P fc469ccaa843461e8421ecfe8b44b210 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:46:40.017541 11127 raft_consensus.cc:504] T f5864b2535544a01bf40662c5ba95f43 P fc469ccaa843461e8421ecfe8b44b210 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe139edec2384aca8046d2de9c18fb99" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 40657 } } peers { permanent_uuid: "fc469ccaa843461e8421ecfe8b44b210" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35295 } } peers { permanent_uuid: "cf3fa56517be4eb8abb73705f5d3a6b6" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44619 } }
I20260810 01:46:40.019182 11127 leader_election.cc:288] T f5864b2535544a01bf40662c5ba95f43 P fc469ccaa843461e8421ecfe8b44b210 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe139edec2384aca8046d2de9c18fb99 (127.9.77.129:40657), cf3fa56517be4eb8abb73705f5d3a6b6 (127.9.77.130:44619)
I20260810 01:46:40.023685 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:40.019578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:44619 (local address 127.9.77.131:59049)
0810 01:46:40.019838 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:40.019860 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:40.019978 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:40.020482 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:40.020484 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:40.020499 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:46:40.020649 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:40.020655 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:40.021488 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:40.021491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:40.022409 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:40.022419 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:40.023103 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:40.023107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:40.023268 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:40.023304 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 01:46:40.023395 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 01:46:40.023494 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":4}
I20260810 01:46:40.023748 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:40.019761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:34883 (local address 127.9.77.129:40657)
0810 01:46:40.019925 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:40.019930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:40.020370 (+ 440us) server_negotiation.cc:407] Connection header received
0810 01:46:40.020459 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:40.020465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:40.020535 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:40.020641 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:46:40.020965 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:40.021494 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:40.022594 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:40.023069 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:40.023208 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:40.023271 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:40.023503 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 01:46:40.023592 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 01:46:40.023800 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:40.019953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:59049 (local address 127.9.77.130:44619)
0810 01:46:40.020139 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:40.020144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:40.020162 (+ 18us) server_negotiation.cc:407] Connection header received
0810 01:46:40.020236 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:40.020241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:40.020338 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:40.020431 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:46:40.020775 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:40.021361 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:40.022545 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:40.022966 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:40.023112 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:40.023185 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:40.023448 (+ 263us) server_negotiation.cc:299] Negotiation successful
0810 01:46:40.023600 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260810 01:46:40.024247 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:40.019577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:40657 (local address 127.9.77.131:34883)
0810 01:46:40.020209 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:40.020225 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:40.020365 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:40.020653 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:40.020656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:40.020674 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:46:40.020829 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:40.020835 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:40.021612 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:40.021618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:40.022454 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:40.022463 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:40.023199 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:40.023202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:40.023349 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:40.023376 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 01:46:40.023939 (+ 563us) client_negotiation.cc:240] Negotiation successful
0810 01:46:40.024056 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":155,"thread_start_us":179,"threads_started":1}
I20260810 01:46:40.025014 10810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5864b2535544a01bf40662c5ba95f43" candidate_uuid: "fc469ccaa843461e8421ecfe8b44b210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf3fa56517be4eb8abb73705f5d3a6b6" is_pre_election: true
I20260810 01:46:40.025347 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5864b2535544a01bf40662c5ba95f43" candidate_uuid: "fc469ccaa843461e8421ecfe8b44b210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe139edec2384aca8046d2de9c18fb99" is_pre_election: true
I20260810 01:46:40.025571 10810 raft_consensus.cc:2454] T f5864b2535544a01bf40662c5ba95f43 P cf3fa56517be4eb8abb73705f5d3a6b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc469ccaa843461e8421ecfe8b44b210 in term 0.
I20260810 01:46:40.025717 10611 raft_consensus.cc:2454] T f5864b2535544a01bf40662c5ba95f43 P fe139edec2384aca8046d2de9c18fb99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc469ccaa843461e8421ecfe8b44b210 in term 0.
I20260810 01:46:40.026342 10918 leader_election.cc:302] T f5864b2535544a01bf40662c5ba95f43 P fc469ccaa843461e8421ecfe8b44b210 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf3fa56517be4eb8abb73705f5d3a6b6, fc469ccaa843461e8421ecfe8b44b210; no voters:
I20260810 01:46:40.026768 11127 raft_consensus.cc:2781] T f5864b2535544a01bf40662c5ba95f43 P fc469ccaa843461e8421ecfe8b44b210 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:46:40.026887 11127 raft_consensus.cc:482] T f5864b2535544a01bf40662c5ba95f43 P fc469ccaa843461e8421ecfe8b44b210 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:46:40.026935 11127 raft_consensus.cc:3037] T f5864b2535544a01bf40662c5ba95f43 P fc469ccaa843461e8421ecfe8b44b210 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:40.028651 11127 raft_consensus.cc:504] T f5864b2535544a01bf40662c5ba95f43 P fc469ccaa843461e8421ecfe8b44b210 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe139edec2384aca8046d2de9c18fb99" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 40657 } } peers { permanent_uuid: "fc469ccaa843461e8421ecfe8b44b210" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35295 } } peers { permanent_uuid: "cf3fa56517be4eb8abb73705f5d3a6b6" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44619 } }
I20260810 01:46:40.029093 11127 leader_election.cc:288] T f5864b2535544a01bf40662c5ba95f43 P fc469ccaa843461e8421ecfe8b44b210 [CANDIDATE]: Term 1 election: Requested vote from peers fe139edec2384aca8046d2de9c18fb99 (127.9.77.129:40657), cf3fa56517be4eb8abb73705f5d3a6b6 (127.9.77.130:44619)
I20260810 01:46:40.029625 10611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5864b2535544a01bf40662c5ba95f43" candidate_uuid: "fc469ccaa843461e8421ecfe8b44b210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe139edec2384aca8046d2de9c18fb99"
I20260810 01:46:40.029718 10810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5864b2535544a01bf40662c5ba95f43" candidate_uuid: "fc469ccaa843461e8421ecfe8b44b210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf3fa56517be4eb8abb73705f5d3a6b6"
I20260810 01:46:40.029805 10611 raft_consensus.cc:3037] T f5864b2535544a01bf40662c5ba95f43 P fe139edec2384aca8046d2de9c18fb99 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:40.029883 10810 raft_consensus.cc:3037] T f5864b2535544a01bf40662c5ba95f43 P cf3fa56517be4eb8abb73705f5d3a6b6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:40.031612 10810 raft_consensus.cc:2454] T f5864b2535544a01bf40662c5ba95f43 P cf3fa56517be4eb8abb73705f5d3a6b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc469ccaa843461e8421ecfe8b44b210 in term 1.
I20260810 01:46:40.031612 10611 raft_consensus.cc:2454] T f5864b2535544a01bf40662c5ba95f43 P fe139edec2384aca8046d2de9c18fb99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc469ccaa843461e8421ecfe8b44b210 in term 1.
I20260810 01:46:40.032153 10915 leader_election.cc:302] T f5864b2535544a01bf40662c5ba95f43 P fc469ccaa843461e8421ecfe8b44b210 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fc469ccaa843461e8421ecfe8b44b210, fe139edec2384aca8046d2de9c18fb99; no voters:
I20260810 01:46:40.032371 11127 raft_consensus.cc:2781] T f5864b2535544a01bf40662c5ba95f43 P fc469ccaa843461e8421ecfe8b44b210 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:46:40.032612 11127 raft_consensus.cc:686] T f5864b2535544a01bf40662c5ba95f43 P fc469ccaa843461e8421ecfe8b44b210 [term 1 LEADER]: Becoming Leader. State: Replica: fc469ccaa843461e8421ecfe8b44b210, State: Running, Role: LEADER
I20260810 01:46:40.033099 11127 consensus_queue.cc:237] T f5864b2535544a01bf40662c5ba95f43 P fc469ccaa843461e8421ecfe8b44b210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe139edec2384aca8046d2de9c18fb99" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 40657 } } peers { permanent_uuid: "fc469ccaa843461e8421ecfe8b44b210" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35295 } } peers { permanent_uuid: "cf3fa56517be4eb8abb73705f5d3a6b6" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44619 } }
I20260810 01:46:40.037385 10382 catalog_manager.cc:5495] T f5864b2535544a01bf40662c5ba95f43 P fc469ccaa843461e8421ecfe8b44b210 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc469ccaa843461e8421ecfe8b44b210 (127.9.77.131). New cstate: current_term: 1 leader_uuid: "fc469ccaa843461e8421ecfe8b44b210" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe139edec2384aca8046d2de9c18fb99" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 40657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc469ccaa843461e8421ecfe8b44b210" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf3fa56517be4eb8abb73705f5d3a6b6" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44619 } health_report { overall_health: UNKNOWN } } }
W20260810 01:46:40.041512 10687 tablet.cc:2369] T f5864b2535544a01bf40662c5ba95f43 P fe139edec2384aca8046d2de9c18fb99: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 01:46:40 dist-test-slave-kvfs krb5kdc[10317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.129@KRBTEST.COM
W20260810 01:46:40.060992 10883 tablet.cc:2369] T f5864b2535544a01bf40662c5ba95f43 P cf3fa56517be4eb8abb73705f5d3a6b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 01:46:40.061396 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:40.050398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32898 (local address 127.9.77.129:40657)
0810 01:46:40.050575 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:40.050582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:40.050798 (+ 216us) server_negotiation.cc:407] Connection header received
0810 01:46:40.050987 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:40.050989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:40.051078 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:40.051182 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:40.052106 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:40.052775 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:40.053715 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:40.053965 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:40.054081 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:40.057054 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:40.057071 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:40.057092 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:40.057122 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:40.059092 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:40.059595 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:40.059602 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:40.059604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:40.059693 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:40.059989 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:40.059993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:40.059995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:40.060385 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:40.060646 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:40.060896 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 01:46:40.061047 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Aug 10 01:46:40 dist-test-slave-kvfs krb5kdc[10317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326400, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260810 01:46:40.079788 11090 tablet.cc:2369] T f5864b2535544a01bf40662c5ba95f43 P fc469ccaa843461e8421ecfe8b44b210: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 01:46:40 dist-test-slave-kvfs krb5kdc[10317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326400, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.77.129@KRBTEST.COM
I20260810 01:46:40.096973 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:40.086437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32906 (local address 127.9.77.129:40657)
0810 01:46:40.086607 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:40.086611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:40.086863 (+ 252us) server_negotiation.cc:407] Connection header received
0810 01:46:40.087002 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:40.087004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:40.087059 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:40.087147 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:40.088108 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:40.088694 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:40.089654 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:40.089922 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:40.090008 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:40.093328 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:40.093331 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:40.093333 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:40.093363 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:40.094876 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:40.095344 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:40.095348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:40.095350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:40.095506 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:40.095800 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:40.095804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:40.095806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:40.096090 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:40.096250 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:40.096608 (+ 358us) server_negotiation.cc:299] Negotiation successful
0810 01:46:40.096759 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 01:46:40 dist-test-slave-kvfs krb5kdc[10317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 01:46:40 dist-test-slave-kvfs krb5kdc[10317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.129@KRBTEST.COM
I20260810 01:46:40.131634 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:40.120832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32922 (local address 127.9.77.129:40657)
0810 01:46:40.121038 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:40.121045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:40.121200 (+ 155us) server_negotiation.cc:407] Connection header received
0810 01:46:40.121439 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:40.121442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:40.121502 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:40.121608 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:40.122477 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:40.123106 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:40.124184 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:40.124454 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:40.124590 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:40.127927 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:40.127930 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:40.127933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:40.127966 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:40.129736 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:40.130237 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:40.130241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:40.130245 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:40.130338 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:40.130644 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:40.130647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:40.130649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:40.130860 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:40.131018 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:40.131274 (+ 256us) server_negotiation.cc:299] Negotiation successful
0810 01:46:40.131414 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260810 01:46:40.133637 10558 tablet_service.cc:2908] Rejecting scan request for tablet f5864b2535544a01bf40662c5ba95f43: Uninitialized: clean time has not yet been initialized
I20260810 01:46:40.136456 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 10494
I20260810 01:46:40.159042 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 10694
I20260810 01:46:40.170642 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 10889
I20260810 01:46:40.186678 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 10338
2026-08-10T01:46:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2962 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:46:40 dist-test-slave-kvfs krb5kdc[11149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:46:40 dist-test-slave-kvfs krb5kdc[11149](info): set up 2 sockets
Aug 10 01:46:40 dist-test-slave-kvfs krb5kdc[11149](info): commencing operation
krb5kdc: starting...
W20260810 01:46:41.514928 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.289s user 0.001s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:46:41 dist-test-slave-kvfs krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:46:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:46:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:46:41.667863 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:45779
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:40413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:45779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:46:41.804198 11167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:46:41.804584 11167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:46:41.804665 11167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:46:41.808863 11167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:46:41.808959 11167 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 01:46:41.808990 11167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:46:41.809031 11167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:46:41.809090 11167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:46:41.809127 11167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:46:41.814718 11167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:40413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:45779
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:45779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:46:41.816190 11167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:46:41.817515 11167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:46:41.827768 11174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:41.827813 11181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:41.828002 11167 server_base.cc:1034] running on GCE node
W20260810 01:46:41.827765 11177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:41.828896 11167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:46:41.830298 11167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:46:41.831532 11167 hybrid_clock.cc:648] HybridClock initialized: now 1786326401831517 us; error 29 us; skew 500 ppm
Aug 10 01:46:41 dist-test-slave-kvfs krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326401, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:46:41.834951 11167 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:46:41.836150 11167 webserver.cc:460] Webserver started at http://127.9.77.190:45989/ using document root <none> and password file <none>
I20260810 01:46:41.836850 11167 fs_manager.cc:359] Metadata directory not provided
I20260810 01:46:41.836949 11167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:46:41.837225 11167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:46:41.839669 11167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "4252c02364924a12a3d85ccadf11e1de"
format_stamp: "Formatted at 2026-08-10 01:46:41 on dist-test-slave-kvfs"
server_key: "91706b182837806fea410066c1a5cde1"
server_key_iv: "e5ac96f1145ace8c104753db0aa3bf3e"
server_key_version: "encryptionkey@0"
I20260810 01:46:41.840355 11167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "4252c02364924a12a3d85ccadf11e1de"
format_stamp: "Formatted at 2026-08-10 01:46:41 on dist-test-slave-kvfs"
server_key: "91706b182837806fea410066c1a5cde1"
server_key_iv: "e5ac96f1145ace8c104753db0aa3bf3e"
server_key_version: "encryptionkey@0"
I20260810 01:46:41.844770 11167 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 01:46:41.847641 11193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:41.849161 11167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:46:41.849328 11167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "4252c02364924a12a3d85ccadf11e1de"
format_stamp: "Formatted at 2026-08-10 01:46:41 on dist-test-slave-kvfs"
server_key: "91706b182837806fea410066c1a5cde1"
server_key_iv: "e5ac96f1145ace8c104753db0aa3bf3e"
server_key_version: "encryptionkey@0"
I20260810 01:46:41.849460 11167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:46:41.869632 11167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:46:41.873442 11167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:46:41.873684 11167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:46:41.884246 11167 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:45779
I20260810 01:46:41.884294 11293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:45779 every 8 connection(s)
I20260810 01:46:41.885469 11167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:46:41.886875 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 11167
I20260810 01:46:41.887022 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:46:41.887396 9526 external_mini_cluster.cc:1428] Setting key bb5a4132021daa45c06b2a4ceb8fe7cb
I20260810 01:46:41.889849 11297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 01:46:41 dist-test-slave-kvfs krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:41.898226 11297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4252c02364924a12a3d85ccadf11e1de: Bootstrap starting.
I20260810 01:46:41.901291 11297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4252c02364924a12a3d85ccadf11e1de: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:41.902640 11297 log.cc:824] T 00000000000000000000000000000000 P 4252c02364924a12a3d85ccadf11e1de: Log is configured to *not* fsync() on all Append() calls
I20260810 01:46:41.904608 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:41.889184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59498 (local address 127.9.77.190:45779)
0810 01:46:41.889687 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:41.889705 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:41.889751 (+ 46us) server_negotiation.cc:407] Connection header received
0810 01:46:41.890591 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:41.890612 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:41.891012 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:41.891356 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:41.892398 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:41.893585 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:41.894823 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:41.895346 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:41.895492 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:41.898929 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:41.898934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:41.898951 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:41.898984 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:41.901004 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:41.901628 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:41.901634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:41.901639 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:41.901769 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:41.902124 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:41.902128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:41.902129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:41.902624 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:41.902877 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:41.903216 (+ 339us) server_negotiation.cc:299] Negotiation successful
0810 01:46:41.903575 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":165,"threads_started":1}
I20260810 01:46:41.905318 11297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4252c02364924a12a3d85ccadf11e1de: No bootstrap required, opened a new log
I20260810 01:46:41.909032 11297 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4252c02364924a12a3d85ccadf11e1de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4252c02364924a12a3d85ccadf11e1de" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45779 } }
I20260810 01:46:41.909286 11297 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4252c02364924a12a3d85ccadf11e1de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:41.909324 11297 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4252c02364924a12a3d85ccadf11e1de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4252c02364924a12a3d85ccadf11e1de, State: Initialized, Role: FOLLOWER
I20260810 01:46:41.909809 11297 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4252c02364924a12a3d85ccadf11e1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4252c02364924a12a3d85ccadf11e1de" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45779 } }
I20260810 01:46:41.909950 11297 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4252c02364924a12a3d85ccadf11e1de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:46:41.910002 11297 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4252c02364924a12a3d85ccadf11e1de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:46:41.910084 11297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4252c02364924a12a3d85ccadf11e1de [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:41.911059 11297 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4252c02364924a12a3d85ccadf11e1de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4252c02364924a12a3d85ccadf11e1de" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45779 } }
I20260810 01:46:41.911410 11297 leader_election.cc:302] T 00000000000000000000000000000000 P 4252c02364924a12a3d85ccadf11e1de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4252c02364924a12a3d85ccadf11e1de; no voters:
I20260810 01:46:41.911751 11297 leader_election.cc:288] T 00000000000000000000000000000000 P 4252c02364924a12a3d85ccadf11e1de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:46:41.911895 11308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4252c02364924a12a3d85ccadf11e1de [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:46:41.912192 11308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4252c02364924a12a3d85ccadf11e1de [term 1 LEADER]: Becoming Leader. State: Replica: 4252c02364924a12a3d85ccadf11e1de, State: Running, Role: LEADER
I20260810 01:46:41.912596 11308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4252c02364924a12a3d85ccadf11e1de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4252c02364924a12a3d85ccadf11e1de" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45779 } }
I20260810 01:46:41.913357 11297 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4252c02364924a12a3d85ccadf11e1de [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:46:41.914649 11310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4252c02364924a12a3d85ccadf11e1de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4252c02364924a12a3d85ccadf11e1de. Latest consensus state: current_term: 1 leader_uuid: "4252c02364924a12a3d85ccadf11e1de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4252c02364924a12a3d85ccadf11e1de" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45779 } } }
I20260810 01:46:41.914793 11310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4252c02364924a12a3d85ccadf11e1de [sys.catalog]: This master's current role is: LEADER
I20260810 01:46:41.915117 11309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4252c02364924a12a3d85ccadf11e1de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4252c02364924a12a3d85ccadf11e1de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4252c02364924a12a3d85ccadf11e1de" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45779 } } }
I20260810 01:46:41.915210 11309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4252c02364924a12a3d85ccadf11e1de [sys.catalog]: This master's current role is: LEADER
I20260810 01:46:41.915331 11323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:46:41.919201 11323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:46:41.927336 11323 catalog_manager.cc:1347] Generated new cluster ID: 4498716e6c854c4c94f154dd66556750
I20260810 01:46:41.927464 11323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:46:41.939579 11323 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:46:41.940891 11323 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:46:41.947172 11323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4252c02364924a12a3d85ccadf11e1de: Generated new TSK 0
I20260810 01:46:41.948274 11323 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.77.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
I20260810 01:46:41.995623 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:0
--local_ip_for_outbound_sockets=127.9.77.129
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:45779
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:40413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:46:42.136302 11337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:46:42.136658 11337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:46:42.136708 11337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:46:42.140947 11337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:46:42.141026 11337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:46:42.141065 11337 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 01:46:42.141136 11337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:46:42.146886 11337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:40413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.77.190:45779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:46:42.148304 11337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:46:42.149597 11337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 01:46:42.161604 11337 server_base.cc:1034] running on GCE node
W20260810 01:46:42.161626 11346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:42.161839 11347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:42.161855 11351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:42.162496 11337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:46:42.163426 11337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:46:42.164614 11337 hybrid_clock.cc:648] HybridClock initialized: now 1786326402164593 us; error 36 us; skew 500 ppm
Aug 10 01:46:42 dist-test-slave-kvfs krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326402, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:46:42.168184 11337 init.cc:377] Logged in from keytab as kudu/127.9.77.129@KRBTEST.COM (short username kudu)
I20260810 01:46:42.169255 11337 webserver.cc:460] Webserver started at http://127.9.77.129:46057/ using document root <none> and password file <none>
I20260810 01:46:42.169965 11337 fs_manager.cc:359] Metadata directory not provided
I20260810 01:46:42.170030 11337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:46:42.170333 11337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:46:42.172933 11337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-0/data/instance:
uuid: "7c410c8eae2f4ca2a445b4038a2da1f8"
format_stamp: "Formatted at 2026-08-10 01:46:42 on dist-test-slave-kvfs"
server_key: "9d458c08089eea347c8dfc5385195b83"
server_key_iv: "927317e9335a5508e0fc99816690e488"
server_key_version: "encryptionkey@0"
I20260810 01:46:42.173673 11337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance:
uuid: "7c410c8eae2f4ca2a445b4038a2da1f8"
format_stamp: "Formatted at 2026-08-10 01:46:42 on dist-test-slave-kvfs"
server_key: "9d458c08089eea347c8dfc5385195b83"
server_key_iv: "927317e9335a5508e0fc99816690e488"
server_key_version: "encryptionkey@0"
I20260810 01:46:42.178102 11337 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 01:46:42.181070 11361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:42.182597 11337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:46:42.182762 11337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "7c410c8eae2f4ca2a445b4038a2da1f8"
format_stamp: "Formatted at 2026-08-10 01:46:42 on dist-test-slave-kvfs"
server_key: "9d458c08089eea347c8dfc5385195b83"
server_key_iv: "927317e9335a5508e0fc99816690e488"
server_key_version: "encryptionkey@0"
I20260810 01:46:42.182904 11337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:46:42.194572 11337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:46:42.198455 11337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:46:42.198690 11337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:46:42.200598 11337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:46:42.200672 11337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:42.200765 11337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:46:42.200806 11337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 01:46:42 dist-test-slave-kvfs krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326402, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:42.213701 11337 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:36033
I20260810 01:46:42.213780 11530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:36033 every 8 connection(s)
I20260810 01:46:42.215287 11337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:46:42.216363 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 11337
I20260810 01:46:42.216513 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance
I20260810 01:46:42.216837 9526 external_mini_cluster.cc:1428] Setting key b76fa62222b4c01e56a7d679af3371a9
I20260810 01:46:42.219949 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:42.201565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:45407 (local address 127.9.77.190:45779)
0810 01:46:42.201731 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:42.201735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:42.202973 (+ 1238us) server_negotiation.cc:407] Connection header received
0810 01:46:42.204377 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:42.204382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:42.204469 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:42.204525 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:42.206284 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.207311 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.208708 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.209152 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.209289 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:42.213849 (+ 4560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:42.213854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:42.213858 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:42.213895 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:42.215948 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:42.217967 (+ 2019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:42.217970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:42.217972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:42.218079 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:42.218513 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:42.218515 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:42.218517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:42.218780 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:42.218871 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:42.219616 (+ 745us) server_negotiation.cc:299] Negotiation successful
0810 01:46:42.219791 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 01:46:42.221453 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:42.202185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45779 (local address 127.9.77.129:45407)
0810 01:46:42.202763 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:42.202819 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:42.204026 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:42.204921 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:42.204928 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:42.205310 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:46:42.206059 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:42.206086 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.207466 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.207469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:42.208563 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:42.208573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.209385 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.209393 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:42.209676 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:42.211289 (+ 1613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:46:42.211324 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:46:42.213544 (+ 2220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:46:42.217420 (+ 3876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:42.217430 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:42.217450 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:42.217819 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:42.218186 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:42.218189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:42.218191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:42.218421 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:42.218895 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:46:42.218902 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:46:42.219299 (+ 397us) client_negotiation.cc:769] Sending connection context
0810 01:46:42.219536 (+ 237us) client_negotiation.cc:240] Negotiation successful
0810 01:46:42.219994 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":165,"threads_started":1}
I20260810 01:46:42.222985 11531 heartbeater.cc:347] Connected to a master server at 127.9.77.190:45779
I20260810 01:46:42.223192 11531 heartbeater.cc:464] Registering TS with master...
I20260810 01:46:42.224061 11531 heartbeater.cc:511] Master 127.9.77.190:45779 requested a full tablet report, sending...
I20260810 01:46:42.226305 11219 ts_manager.cc:194] Registered new tserver with Master: 7c410c8eae2f4ca2a445b4038a2da1f8 (127.9.77.129:36033)
I20260810 01:46:42.228688 11219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.129@KRBTEST.COM'} at 127.9.77.129:45407
WARNING: no policy specified for kudu/127.9.77.130@KRBTEST.COM; defaulting to no policy
I20260810 01:46:42.234596 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:42.226301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:53417 (local address 127.9.77.190:45779)
0810 01:46:42.226445 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:42.226450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:42.226724 (+ 274us) server_negotiation.cc:407] Connection header received
0810 01:46:42.226891 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:42.226894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:42.226949 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:42.227046 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:42.227593 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.228395 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.229384 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.229718 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.229832 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:42.231138 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:42.231142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:42.231145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:42.231172 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:42.232861 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:42.233303 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:42.233307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:42.233309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:42.233397 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:42.233689 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:42.233694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:42.233695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:42.233884 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:42.234001 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:42.234265 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 01:46:42.234401 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":126,"server-negotiator.queue_time_us":60}
I20260810 01:46:42.234597 11542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:42.226233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45779 (local address 127.9.77.129:53417)
0810 01:46:42.226615 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:42.226633 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:42.226772 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:42.227062 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:42.227065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:42.227333 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:46:42.227466 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:42.227472 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.228526 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.228530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:42.229251 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:42.229260 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.229839 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.229843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:42.229943 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:42.230596 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:46:42.230609 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:46:42.230991 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:46:42.232973 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:42.232977 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:42.232979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:42.233197 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:42.233497 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:42.233501 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:42.233503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:42.233596 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:42.233995 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:46:42.233998 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:46:42.234099 (+ 101us) client_negotiation.cc:769] Sending connection context
0810 01:46:42.234209 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 01:46:42.234371 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":124,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
I20260810 01:46:42.254596 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:0
--local_ip_for_outbound_sockets=127.9.77.130
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:45779
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:40413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:46:42.393347 11544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:46:42.393723 11544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:46:42.393812 11544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:46:42.397991 11544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:46:42.398095 11544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:46:42.398173 11544 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 01:46:42.398260 11544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:46:42.403939 11544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:40413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.77.190:45779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:46:42.405427 11544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:46:42.406766 11544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:46:42.418576 11553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:42.418617 11544 server_base.cc:1034] running on GCE node
W20260810 01:46:42.418545 11557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:42.418535 11552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:42.419296 11544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:46:42.420104 11544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:46:42.421293 11544 hybrid_clock.cc:648] HybridClock initialized: now 1786326402421272 us; error 30 us; skew 500 ppm
Aug 10 01:46:42 dist-test-slave-kvfs krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326402, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:46:42.424860 11544 init.cc:377] Logged in from keytab as kudu/127.9.77.130@KRBTEST.COM (short username kudu)
I20260810 01:46:42.425923 11544 webserver.cc:460] Webserver started at http://127.9.77.130:37965/ using document root <none> and password file <none>
I20260810 01:46:42.426581 11544 fs_manager.cc:359] Metadata directory not provided
I20260810 01:46:42.426643 11544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:46:42.426846 11544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:46:42.429417 11544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-1/data/instance:
uuid: "906d160bda324389b8c184d19cf04eb4"
format_stamp: "Formatted at 2026-08-10 01:46:42 on dist-test-slave-kvfs"
server_key: "fea103066e4184933072de8044de4c95"
server_key_iv: "a8178fa96f7a5acc7a22712a8d4fb3be"
server_key_version: "encryptionkey@0"
I20260810 01:46:42.430023 11544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance:
uuid: "906d160bda324389b8c184d19cf04eb4"
format_stamp: "Formatted at 2026-08-10 01:46:42 on dist-test-slave-kvfs"
server_key: "fea103066e4184933072de8044de4c95"
server_key_iv: "a8178fa96f7a5acc7a22712a8d4fb3be"
server_key_version: "encryptionkey@0"
I20260810 01:46:42.434737 11544 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 01:46:42.437736 11566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:42.439270 11544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 01:46:42.439424 11544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "906d160bda324389b8c184d19cf04eb4"
format_stamp: "Formatted at 2026-08-10 01:46:42 on dist-test-slave-kvfs"
server_key: "fea103066e4184933072de8044de4c95"
server_key_iv: "a8178fa96f7a5acc7a22712a8d4fb3be"
server_key_version: "encryptionkey@0"
I20260810 01:46:42.439556 11544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:46:42.464327 11544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:46:42.468609 11544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:46:42.468868 11544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:46:42.470773 11544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:46:42.470860 11544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:42.470933 11544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:46:42.470978 11544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 01:46:42 dist-test-slave-kvfs krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326402, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:42.484128 11544 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:45649
I20260810 01:46:42.484258 11744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:45649 every 8 connection(s)
I20260810 01:46:42.485546 11544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:46:42.487896 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:42.472011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:47887 (local address 127.9.77.190:45779)
0810 01:46:42.472222 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:42.472228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:42.473596 (+ 1368us) server_negotiation.cc:407] Connection header received
0810 01:46:42.475219 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:42.475228 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:42.475325 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:42.475451 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:42.477168 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.478198 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.479659 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.479935 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.480105 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:42.483275 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:42.483279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:42.483283 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:42.483328 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:42.485335 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:42.485960 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:42.485965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:42.485969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:42.486090 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:42.486506 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:42.486510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:42.486512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:42.486807 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:42.486964 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:42.487535 (+ 571us) server_negotiation.cc:299] Negotiation successful
0810 01:46:42.487696 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 01:46:42.488883 11579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:42.472706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45779 (local address 127.9.77.130:47887)
0810 01:46:42.473393 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:42.473447 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:42.474886 (+ 1439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:42.475674 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:42.475682 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:42.476312 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:46:42.476994 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:42.477012 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.478346 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.478349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:42.479504 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:42.479515 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.480142 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.480148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:42.480379 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:42.481168 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:46:42.481189 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:46:42.483023 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:46:42.485466 (+ 2443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:42.485471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:42.485484 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:42.485843 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:42.486213 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:42.486216 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:42.486218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:42.486414 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:42.486956 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:46:42.486963 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:46:42.487253 (+ 290us) client_negotiation.cc:769] Sending connection context
0810 01:46:42.487490 (+ 237us) client_negotiation.cc:240] Negotiation successful
0810 01:46:42.487867 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":180,"threads_started":1}
I20260810 01:46:42.490286 11745 heartbeater.cc:347] Connected to a master server at 127.9.77.190:45779
I20260810 01:46:42.490448 11745 heartbeater.cc:464] Registering TS with master...
I20260810 01:46:42.491166 11745 heartbeater.cc:511] Master 127.9.77.190:45779 requested a full tablet report, sending...
I20260810 01:46:42.492647 11220 ts_manager.cc:194] Registered new tserver with Master: 906d160bda324389b8c184d19cf04eb4 (127.9.77.130:45649)
I20260810 01:46:42.493517 11220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.130@KRBTEST.COM'} at 127.9.77.130:47887
I20260810 01:46:42.493845 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 11544
I20260810 01:46:42.493974 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance
I20260810 01:46:42.494362 9526 external_mini_cluster.cc:1428] Setting key d48b292c446baeb91a58f4aa6ef466bf
I20260810 01:46:42.501789 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:42.493371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:34565 (local address 127.9.77.190:45779)
0810 01:46:42.493526 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:42.493531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:42.493687 (+ 156us) server_negotiation.cc:407] Connection header received
0810 01:46:42.493923 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:42.493926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:42.493977 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:42.494034 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:42.494706 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.495442 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.496517 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.496893 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.497022 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:42.498230 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:42.498238 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:42.498244 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:42.498279 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:42.499937 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:42.500479 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:42.500481 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:42.500482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:42.500563 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:42.500890 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:42.500892 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:42.500893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:42.501129 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:42.501212 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:42.501532 (+ 320us) server_negotiation.cc:299] Negotiation successful
0810 01:46:42.501642 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 01:46:42.501960 11752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:42.493220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45779 (local address 127.9.77.130:34565)
0810 01:46:42.493588 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:42.493607 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:42.493770 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:42.494186 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:42.494189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:42.494447 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:46:42.494587 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:42.494593 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.495586 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.495590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:42.496385 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:42.496394 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.497024 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.497026 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:42.497102 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:42.497701 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:46:42.497712 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:46:42.498079 (+ 367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:46:42.500084 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:42.500090 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:42.500093 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:42.500379 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:42.500660 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:42.500665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:42.500670 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:42.500785 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:42.501230 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:46:42.501236 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:46:42.501362 (+ 126us) client_negotiation.cc:769] Sending connection context
0810 01:46:42.501480 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 01:46:42.501669 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for kudu/127.9.77.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
I20260810 01:46:42.528091 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:0
--local_ip_for_outbound_sockets=127.9.77.131
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:45779
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:40413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:46:42.663277 11755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:46:42.663653 11755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:46:42.663703 11755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:46:42.667968 11755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:46:42.668054 11755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:46:42.668095 11755 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 01:46:42.668130 11755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:46:42.674149 11755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:40413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.77.190:45779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:46:42.675510 11755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:46:42.676901 11755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:46:42.688356 11762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:42.688376 11763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:42.688408 11765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:42.688797 11755 server_base.cc:1034] running on GCE node
I20260810 01:46:42.689256 11755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:46:42.690028 11755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:46:42.691264 11755 hybrid_clock.cc:648] HybridClock initialized: now 1786326402691247 us; error 32 us; skew 500 ppm
Aug 10 01:46:42 dist-test-slave-kvfs krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326402, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:46:42.694697 11755 init.cc:377] Logged in from keytab as kudu/127.9.77.131@KRBTEST.COM (short username kudu)
I20260810 01:46:42.695711 11755 webserver.cc:460] Webserver started at http://127.9.77.131:33603/ using document root <none> and password file <none>
I20260810 01:46:42.696436 11755 fs_manager.cc:359] Metadata directory not provided
I20260810 01:46:42.696506 11755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:46:42.696717 11755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:46:42.699141 11755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-2/data/instance:
uuid: "208e6a0e5f224cf9ac3776e3fa28819b"
format_stamp: "Formatted at 2026-08-10 01:46:42 on dist-test-slave-kvfs"
server_key: "ffe5082ed992f3890da248bf7db17332"
server_key_iv: "5ff90d053b5192b85c8fbeaaef8f785d"
server_key_version: "encryptionkey@0"
I20260810 01:46:42.699724 11755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance:
uuid: "208e6a0e5f224cf9ac3776e3fa28819b"
format_stamp: "Formatted at 2026-08-10 01:46:42 on dist-test-slave-kvfs"
server_key: "ffe5082ed992f3890da248bf7db17332"
server_key_iv: "5ff90d053b5192b85c8fbeaaef8f785d"
server_key_version: "encryptionkey@0"
I20260810 01:46:42.704255 11755 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.004s
I20260810 01:46:42.707252 11772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:42.708954 11755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:46:42.709103 11755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "208e6a0e5f224cf9ac3776e3fa28819b"
format_stamp: "Formatted at 2026-08-10 01:46:42 on dist-test-slave-kvfs"
server_key: "ffe5082ed992f3890da248bf7db17332"
server_key_iv: "5ff90d053b5192b85c8fbeaaef8f785d"
server_key_version: "encryptionkey@0"
I20260810 01:46:42.709237 11755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:46:42.727221 11755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:46:42.731133 11755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:46:42.731389 11755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:46:42.733373 11755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:46:42.733484 11755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:42.733588 11755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:46:42.733615 11755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 01:46:42 dist-test-slave-kvfs krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326402, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:42.746685 11755 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:33947
I20260810 01:46:42.746722 11954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:33947 every 8 connection(s)
I20260810 01:46:42.747998 11755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:46:42.751370 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:42.734265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:40597 (local address 127.9.77.190:45779)
0810 01:46:42.734440 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:42.734444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:42.736453 (+ 2009us) server_negotiation.cc:407] Connection header received
0810 01:46:42.737864 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:42.737868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:42.737965 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:42.738077 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:42.740608 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.741292 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.743320 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.743565 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.743655 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:42.747355 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:42.747360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:42.747363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:42.747399 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:42.749225 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:42.749764 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:42.749768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:42.749769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:42.749877 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:42.750265 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:42.750268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:42.750269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:42.750479 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:42.750559 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:42.751089 (+ 530us) server_negotiation.cc:299] Negotiation successful
0810 01:46:42.751219 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 01:46:42.752585 11785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:42.735630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45779 (local address 127.9.77.131:40597)
0810 01:46:42.736241 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:42.736291 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:42.737597 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:42.738734 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:42.738745 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:42.739405 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:46:42.740394 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:42.740423 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.741415 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.741422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:42.743183 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:42.743194 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.743778 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.743784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:42.744102 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:42.744978 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:46:42.745005 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:46:42.747151 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:46:42.749369 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:42.749373 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:42.749386 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:42.749654 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:42.749973 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:42.749979 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:42.749984 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:42.750169 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:42.750570 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:46:42.750576 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:46:42.750838 (+ 262us) client_negotiation.cc:769] Sending connection context
0810 01:46:42.751018 (+ 180us) client_negotiation.cc:240] Negotiation successful
0810 01:46:42.751317 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":165,"threads_started":1}
I20260810 01:46:42.754104 11957 heartbeater.cc:347] Connected to a master server at 127.9.77.190:45779
I20260810 01:46:42.754300 11957 heartbeater.cc:464] Registering TS with master...
I20260810 01:46:42.755004 11957 heartbeater.cc:511] Master 127.9.77.190:45779 requested a full tablet report, sending...
I20260810 01:46:42.756417 11219 ts_manager.cc:194] Registered new tserver with Master: 208e6a0e5f224cf9ac3776e3fa28819b (127.9.77.131:33947)
I20260810 01:46:42.757299 11219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.131@KRBTEST.COM'} at 127.9.77.131:40597
I20260810 01:46:42.757387 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 11755
I20260810 01:46:42.757561 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance
I20260810 01:46:42.757918 9526 external_mini_cluster.cc:1428] Setting key d5cf2204f3b8d9a327886295579b5918
I20260810 01:46:42.761710 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 01:46:42.765312 11968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:42.757003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45779 (local address 127.9.77.131:53169)
0810 01:46:42.757777 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:42.757793 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:42.757905 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:42.758350 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:42.758353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:42.758565 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:46:42.758672 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:42.758678 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.759481 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.759485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:42.760142 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:42.760152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.760628 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.760631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:42.760732 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:42.761314 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:46:42.761326 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:46:42.761788 (+ 462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:46:42.763639 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:42.763642 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:42.763645 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:42.763855 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:42.764237 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:42.764240 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:42.764242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:42.764336 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:42.764827 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:46:42.764830 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:46:42.764934 (+ 104us) client_negotiation.cc:769] Sending connection context
0810 01:46:42.765027 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 01:46:42.765150 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":120,"threads_started":1}
I20260810 01:46:42.765486 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:42.757070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:53169 (local address 127.9.77.190:45779)
0810 01:46:42.757200 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:42.757204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:42.757885 (+ 681us) server_negotiation.cc:407] Connection header received
0810 01:46:42.757993 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:42.757997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:42.758052 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:42.758161 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:42.758793 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.759345 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.760264 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.760508 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.760622 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:42.761928 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:42.761933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:42.761936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:42.761965 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:42.763511 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:42.763976 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:42.763980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:42.763984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:42.764126 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:42.764439 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:42.764442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:42.764444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:42.764715 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:42.764828 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:42.765114 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 01:46:42.765262 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 01:46:42 dist-test-slave-kvfs krb5kdc[11149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326402, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 01:46:42 dist-test-slave-kvfs krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326402, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:42.794386 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:42.784863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59502 (local address 127.9.77.190:45779)
0810 01:46:42.785032 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:42.785035 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:42.785049 (+ 14us) server_negotiation.cc:407] Connection header received
0810 01:46:42.785103 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:42.785105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:42.785151 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:42.785254 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:42.786064 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.786711 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.787721 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.788000 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.788169 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:42.790940 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:42.790962 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:42.790965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:42.790999 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:42.792565 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:42.793021 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:42.793025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:42.793029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:42.793116 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:42.793403 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:42.793407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:42.793408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:42.793621 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:42.793779 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:42.794049 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 01:46:42.794183 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 01:46:42.798060 11219 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 01:46:42.802066 11219 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:46:42.818913 11984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:42.812180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:45649 (local address 127.0.0.1:60666)
0810 01:46:42.812667 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:42.812732 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:42.812909 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:42.814012 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:42.814021 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:42.814052 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:46:42.814236 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:42.814243 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.815814 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.815817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:42.817160 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:42.817170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.818263 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.818266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:42.818406 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:42.818453 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 01:46:42.818584 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 01:46:42.818738 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":148,"threads_started":1}
I20260810 01:46:42.818928 11985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:42.812354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60666 (local address 127.9.77.130:45649)
0810 01:46:42.813079 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:42.813092 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:42.813120 (+ 28us) server_negotiation.cc:407] Connection header received
0810 01:46:42.813240 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:42.813247 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:42.813527 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:42.813711 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:46:42.814385 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.815679 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.817310 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.818122 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.818262 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:42.818516 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:42.818618 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 01:46:42.818728 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":163,"threads_started":1}
I20260810 01:46:42.819149 11988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:42.813561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:36033 (local address 127.0.0.1:53662)
0810 01:46:42.813883 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:42.813896 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:42.814009 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:42.814843 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:42.814848 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:42.814869 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:46:42.814967 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:42.814975 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.816631 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.816637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:42.817543 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:42.817555 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.818722 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.818726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:42.818819 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:42.818846 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 01:46:42.818897 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 01:46:42.818981 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":110,"threads_started":1}
I20260810 01:46:42.819856 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:42.813986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53662 (local address 127.9.77.129:36033)
0810 01:46:42.814313 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:42.814320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:42.814344 (+ 24us) server_negotiation.cc:407] Connection header received
0810 01:46:42.814456 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:42.814463 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:42.814676 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:42.815130 (+ 454us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:46:42.815292 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.816467 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.817698 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.818568 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.819335 (+ 767us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:42.819529 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:42.819583 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 01:46:42.819703 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":140,"threads_started":1}
I20260810 01:46:42.820382 11986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:42.813143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:33947 (local address 127.0.0.1:51760)
0810 01:46:42.813395 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:42.813411 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:42.813516 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:42.814545 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:42.814548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:42.814568 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:46:42.814748 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:42.814760 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.816581 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.816585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:42.817575 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:42.817583 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.819762 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.819766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:42.819875 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:42.819900 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 01:46:42.820165 (+ 265us) client_negotiation.cc:240] Negotiation successful
0810 01:46:42.820257 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":118,"threads_started":1}
I20260810 01:46:42.820582 11987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:42.813344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51760 (local address 127.9.77.131:33947)
0810 01:46:42.813997 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:42.814012 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:42.814036 (+ 24us) server_negotiation.cc:407] Connection header received
0810 01:46:42.814123 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:42.814128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:42.814343 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:42.814556 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:46:42.814894 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.816447 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.817735 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:42.819628 (+ 1893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:42.819739 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:42.819912 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:42.820221 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 01:46:42.820328 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":130,"threads_started":1}
I20260810 01:46:42.823271 11854 tablet_service.cc:1467] Processing CreateTablet for tablet e388fb22d81e4b08a71710cc04adfcc0 (DEFAULT_TABLE table=test-table [id=c27f33b65eb14c4a828d7ea72a695767]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:46:42.823148 11640 tablet_service.cc:1467] Processing CreateTablet for tablet e388fb22d81e4b08a71710cc04adfcc0 (DEFAULT_TABLE table=test-table [id=c27f33b65eb14c4a828d7ea72a695767]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:46:42.823130 11433 tablet_service.cc:1467] Processing CreateTablet for tablet e388fb22d81e4b08a71710cc04adfcc0 (DEFAULT_TABLE table=test-table [id=c27f33b65eb14c4a828d7ea72a695767]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:46:42.824585 11854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e388fb22d81e4b08a71710cc04adfcc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:42.824589 11640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e388fb22d81e4b08a71710cc04adfcc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:42.824589 11433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e388fb22d81e4b08a71710cc04adfcc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:42.833819 11993 tablet_bootstrap.cc:492] T e388fb22d81e4b08a71710cc04adfcc0 P 208e6a0e5f224cf9ac3776e3fa28819b: Bootstrap starting.
I20260810 01:46:42.834161 11995 tablet_bootstrap.cc:492] T e388fb22d81e4b08a71710cc04adfcc0 P 906d160bda324389b8c184d19cf04eb4: Bootstrap starting.
I20260810 01:46:42.834811 11994 tablet_bootstrap.cc:492] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8: Bootstrap starting.
I20260810 01:46:42.836653 11995 tablet_bootstrap.cc:654] T e388fb22d81e4b08a71710cc04adfcc0 P 906d160bda324389b8c184d19cf04eb4: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:42.836663 11993 tablet_bootstrap.cc:654] T e388fb22d81e4b08a71710cc04adfcc0 P 208e6a0e5f224cf9ac3776e3fa28819b: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:42.837023 11994 tablet_bootstrap.cc:654] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:42.837666 11995 log.cc:824] T e388fb22d81e4b08a71710cc04adfcc0 P 906d160bda324389b8c184d19cf04eb4: Log is configured to *not* fsync() on all Append() calls
I20260810 01:46:42.837671 11993 log.cc:824] T e388fb22d81e4b08a71710cc04adfcc0 P 208e6a0e5f224cf9ac3776e3fa28819b: Log is configured to *not* fsync() on all Append() calls
I20260810 01:46:42.837843 11994 log.cc:824] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8: Log is configured to *not* fsync() on all Append() calls
I20260810 01:46:42.839921 11995 tablet_bootstrap.cc:492] T e388fb22d81e4b08a71710cc04adfcc0 P 906d160bda324389b8c184d19cf04eb4: No bootstrap required, opened a new log
I20260810 01:46:42.839926 11993 tablet_bootstrap.cc:492] T e388fb22d81e4b08a71710cc04adfcc0 P 208e6a0e5f224cf9ac3776e3fa28819b: No bootstrap required, opened a new log
I20260810 01:46:42.840194 11993 ts_tablet_manager.cc:1397] T e388fb22d81e4b08a71710cc04adfcc0 P 208e6a0e5f224cf9ac3776e3fa28819b: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 01:46:42.840194 11995 ts_tablet_manager.cc:1397] T e388fb22d81e4b08a71710cc04adfcc0 P 906d160bda324389b8c184d19cf04eb4: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260810 01:46:42.842516 11994 tablet_bootstrap.cc:492] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8: No bootstrap required, opened a new log
I20260810 01:46:42.842739 11994 ts_tablet_manager.cc:1397] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260810 01:46:42.844334 11995 raft_consensus.cc:348] T e388fb22d81e4b08a71710cc04adfcc0 P 906d160bda324389b8c184d19cf04eb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } } peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } }
I20260810 01:46:42.844659 11995 raft_consensus.cc:374] T e388fb22d81e4b08a71710cc04adfcc0 P 906d160bda324389b8c184d19cf04eb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:42.844728 11995 raft_consensus.cc:729] T e388fb22d81e4b08a71710cc04adfcc0 P 906d160bda324389b8c184d19cf04eb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 906d160bda324389b8c184d19cf04eb4, State: Initialized, Role: FOLLOWER
I20260810 01:46:42.844626 11993 raft_consensus.cc:348] T e388fb22d81e4b08a71710cc04adfcc0 P 208e6a0e5f224cf9ac3776e3fa28819b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } } peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } }
I20260810 01:46:42.844894 11993 raft_consensus.cc:374] T e388fb22d81e4b08a71710cc04adfcc0 P 208e6a0e5f224cf9ac3776e3fa28819b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:42.844969 11993 raft_consensus.cc:729] T e388fb22d81e4b08a71710cc04adfcc0 P 208e6a0e5f224cf9ac3776e3fa28819b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 208e6a0e5f224cf9ac3776e3fa28819b, State: Initialized, Role: FOLLOWER
I20260810 01:46:42.845306 11995 consensus_queue.cc:260] T e388fb22d81e4b08a71710cc04adfcc0 P 906d160bda324389b8c184d19cf04eb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } } peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } }
I20260810 01:46:42.845404 11993 consensus_queue.cc:260] T e388fb22d81e4b08a71710cc04adfcc0 P 208e6a0e5f224cf9ac3776e3fa28819b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } } peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } }
I20260810 01:46:42.846159 11745 heartbeater.cc:503] Master 127.9.77.190:45779 was elected leader, sending a full tablet report...
I20260810 01:46:42.846477 11957 heartbeater.cc:503] Master 127.9.77.190:45779 was elected leader, sending a full tablet report...
I20260810 01:46:42.846611 11993 ts_tablet_manager.cc:1428] T e388fb22d81e4b08a71710cc04adfcc0 P 208e6a0e5f224cf9ac3776e3fa28819b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 01:46:42.847432 11995 ts_tablet_manager.cc:1428] T e388fb22d81e4b08a71710cc04adfcc0 P 906d160bda324389b8c184d19cf04eb4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 01:46:42.847836 11994 raft_consensus.cc:348] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } } peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } }
I20260810 01:46:42.848129 11994 raft_consensus.cc:374] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:42.848188 11994 raft_consensus.cc:729] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c410c8eae2f4ca2a445b4038a2da1f8, State: Initialized, Role: FOLLOWER
I20260810 01:46:42.848657 11994 consensus_queue.cc:260] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } } peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } }
I20260810 01:46:42.849444 11531 heartbeater.cc:503] Master 127.9.77.190:45779 was elected leader, sending a full tablet report...
I20260810 01:46:42.849686 11994 ts_tablet_manager.cc:1428] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
W20260810 01:46:42.967097 11532 tablet.cc:2369] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 01:46:42.987052 11746 tablet.cc:2369] T e388fb22d81e4b08a71710cc04adfcc0 P 906d160bda324389b8c184d19cf04eb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 01:46:42.999390 11958 tablet.cc:2369] T e388fb22d81e4b08a71710cc04adfcc0 P 208e6a0e5f224cf9ac3776e3fa28819b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 01:46:43.075637 12003 raft_consensus.cc:482] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:46:43.076084 12003 raft_consensus.cc:504] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } } peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } }
I20260810 01:46:43.077793 12003 leader_election.cc:288] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 906d160bda324389b8c184d19cf04eb4 (127.9.77.130:45649), 208e6a0e5f224cf9ac3776e3fa28819b (127.9.77.131:33947)
I20260810 01:46:43.082690 12005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:43.078764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:33947 (local address 127.9.77.129:53901)
0810 01:46:43.079136 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:43.079150 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:43.079285 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:43.079624 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:43.079627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:43.079644 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:46:43.079787 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:43.079792 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:43.080598 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:43.080604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:43.081472 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:43.081481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:43.082202 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:43.082206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:43.082300 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:43.082333 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 01:46:43.082434 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 01:46:43.082512 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":118,"threads_started":1}
I20260810 01:46:43.082674 11987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:43.079009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:53901 (local address 127.9.77.131:33947)
0810 01:46:43.079150 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:43.079155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:43.079251 (+ 96us) server_negotiation.cc:407] Connection header received
0810 01:46:43.079458 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:43.079461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:43.079514 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:43.079610 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:46:43.079913 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:43.080482 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:43.081601 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:43.082078 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:43.082211 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:43.082301 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:43.082467 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 01:46:43.082544 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 01:46:43.082983 11985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:43.078340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:46439 (local address 127.9.77.130:45649)
0810 01:46:43.078659 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:43.078666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:43.078921 (+ 255us) server_negotiation.cc:407] Connection header received
0810 01:46:43.079151 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:43.079154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:43.079215 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:43.079266 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:46:43.079735 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:43.080501 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:43.081672 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:43.082314 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:43.082475 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:43.082541 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:43.082735 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 01:46:43.082830 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 01:46:43.083032 12004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:43.078180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:45649 (local address 127.9.77.129:46439)
0810 01:46:43.078742 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:43.078758 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:43.078956 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:43.079433 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:43.079435 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:43.079453 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:46:43.079576 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:43.079581 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:43.080639 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:43.080646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:43.081527 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:43.081536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:43.082458 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:43.082462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:43.082585 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:43.082612 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 01:46:43.082712 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 01:46:43.082788 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":187,"threads_started":1}
I20260810 01:46:43.083735 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e388fb22d81e4b08a71710cc04adfcc0" candidate_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" is_pre_election: true
I20260810 01:46:43.083729 11671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e388fb22d81e4b08a71710cc04adfcc0" candidate_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "906d160bda324389b8c184d19cf04eb4" is_pre_election: true
I20260810 01:46:43.084163 11887 raft_consensus.cc:2454] T e388fb22d81e4b08a71710cc04adfcc0 P 208e6a0e5f224cf9ac3776e3fa28819b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c410c8eae2f4ca2a445b4038a2da1f8 in term 0.
I20260810 01:46:43.084163 11671 raft_consensus.cc:2454] T e388fb22d81e4b08a71710cc04adfcc0 P 906d160bda324389b8c184d19cf04eb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c410c8eae2f4ca2a445b4038a2da1f8 in term 0.
I20260810 01:46:43.084766 11363 leader_election.cc:302] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c410c8eae2f4ca2a445b4038a2da1f8, 906d160bda324389b8c184d19cf04eb4; no voters:
I20260810 01:46:43.085124 12003 raft_consensus.cc:2781] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:46:43.085218 12003 raft_consensus.cc:482] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:46:43.085289 12003 raft_consensus.cc:3037] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:43.086688 12003 raft_consensus.cc:504] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } } peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } }
I20260810 01:46:43.087074 12003 leader_election.cc:288] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8 [CANDIDATE]: Term 1 election: Requested vote from peers 906d160bda324389b8c184d19cf04eb4 (127.9.77.130:45649), 208e6a0e5f224cf9ac3776e3fa28819b (127.9.77.131:33947)
I20260810 01:46:43.087504 11671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e388fb22d81e4b08a71710cc04adfcc0" candidate_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "906d160bda324389b8c184d19cf04eb4"
I20260810 01:46:43.087541 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e388fb22d81e4b08a71710cc04adfcc0" candidate_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "208e6a0e5f224cf9ac3776e3fa28819b"
I20260810 01:46:43.087680 11671 raft_consensus.cc:3037] T e388fb22d81e4b08a71710cc04adfcc0 P 906d160bda324389b8c184d19cf04eb4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:43.087704 11887 raft_consensus.cc:3037] T e388fb22d81e4b08a71710cc04adfcc0 P 208e6a0e5f224cf9ac3776e3fa28819b [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:43.088953 11671 raft_consensus.cc:2454] T e388fb22d81e4b08a71710cc04adfcc0 P 906d160bda324389b8c184d19cf04eb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c410c8eae2f4ca2a445b4038a2da1f8 in term 1.
I20260810 01:46:43.089290 11363 leader_election.cc:302] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c410c8eae2f4ca2a445b4038a2da1f8, 906d160bda324389b8c184d19cf04eb4; no voters:
I20260810 01:46:43.089505 12003 raft_consensus.cc:2781] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:46:43.089763 12003 raft_consensus.cc:686] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 1 LEADER]: Becoming Leader. State: Replica: 7c410c8eae2f4ca2a445b4038a2da1f8, State: Running, Role: LEADER
I20260810 01:46:43.089821 11887 raft_consensus.cc:2454] T e388fb22d81e4b08a71710cc04adfcc0 P 208e6a0e5f224cf9ac3776e3fa28819b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c410c8eae2f4ca2a445b4038a2da1f8 in term 1.
I20260810 01:46:43.090248 12003 consensus_queue.cc:237] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } } peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } }
I20260810 01:46:43.094116 11219 catalog_manager.cc:5495] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c410c8eae2f4ca2a445b4038a2da1f8 (127.9.77.129). New cstate: current_term: 1 leader_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } health_report { overall_health: HEALTHY } } }
I20260810 01:46:43.121613 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:43.117415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53670 (local address 127.9.77.129:36033)
0810 01:46:43.117569 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:43.117573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:43.117590 (+ 17us) server_negotiation.cc:407] Connection header received
0810 01:46:43.117847 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:43.117851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:43.117926 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:43.118026 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:46:43.118303 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:43.118891 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:43.119869 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:43.120133 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:43.120236 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:43.120562 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:46:43.121153 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:46:43.121253 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:43.121382 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 01:46:43.121462 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 01:46:43.129410 11887 raft_consensus.cc:1261] T e388fb22d81e4b08a71710cc04adfcc0 P 208e6a0e5f224cf9ac3776e3fa28819b [term 1 FOLLOWER]: Refusing update from remote peer 7c410c8eae2f4ca2a445b4038a2da1f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:46:43.129410 11671 raft_consensus.cc:1261] T e388fb22d81e4b08a71710cc04adfcc0 P 906d160bda324389b8c184d19cf04eb4 [term 1 FOLLOWER]: Refusing update from remote peer 7c410c8eae2f4ca2a445b4038a2da1f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:46:43.130255 12006 consensus_queue.cc:1035] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:46:43.130506 12003 consensus_queue.cc:1035] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:46:43.141068 12012 mvcc.cc:205] Tried to move back new op lower bound from 7316792947208228864 to 7316792947060703232. Current Snapshot: MvccSnapshot[applied={T|T < 7316792947208228864}]
I20260810 01:46:43.143471 12009 mvcc.cc:205] Tried to move back new op lower bound from 7316792947208228864 to 7316792947060703232. Current Snapshot: MvccSnapshot[applied={T|T < 7316792947208228864 or (T in {7316792947208228864})}]
I20260810 01:46:43.155943 11218 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:46:43.156313 11218 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:46:43.159653 11218 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4252c02364924a12a3d85ccadf11e1de: Sending DeleteTablet for 3 replicas of tablet e388fb22d81e4b08a71710cc04adfcc0
I20260810 01:46:43.160746 11640 tablet_service.cc:1514] Processing DeleteTablet for tablet e388fb22d81e4b08a71710cc04adfcc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:46:43 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:60666
I20260810 01:46:43.160979 11854 tablet_service.cc:1514] Processing DeleteTablet for tablet e388fb22d81e4b08a71710cc04adfcc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:46:43 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:51760
I20260810 01:46:43.161556 12026 tablet_replica.cc:330] T e388fb22d81e4b08a71710cc04adfcc0 P 208e6a0e5f224cf9ac3776e3fa28819b: stopping tablet replica
I20260810 01:46:43.161545 11433 tablet_service.cc:1514] Processing DeleteTablet for tablet e388fb22d81e4b08a71710cc04adfcc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:46:43 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:53662
I20260810 01:46:43.161672 11219 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 01:46:43.162143 12026 raft_consensus.cc:2229] T e388fb22d81e4b08a71710cc04adfcc0 P 208e6a0e5f224cf9ac3776e3fa28819b [term 1 FOLLOWER]: Raft consensus shutting down.
W20260810 01:46:43.162189 11219 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:46:43.162267 12027 tablet_replica.cc:330] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8: stopping tablet replica
I20260810 01:46:43.162618 12027 raft_consensus.cc:2229] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 1 LEADER]: Raft consensus shutting down.
I20260810 01:46:43.162784 12026 raft_consensus.cc:2258] T e388fb22d81e4b08a71710cc04adfcc0 P 208e6a0e5f224cf9ac3776e3fa28819b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:46:43.163127 12027 raft_consensus.cc:2258] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:46:43.163219 12025 tablet_replica.cc:330] T e388fb22d81e4b08a71710cc04adfcc0 P 906d160bda324389b8c184d19cf04eb4: stopping tablet replica
I20260810 01:46:43.163559 12025 raft_consensus.cc:2229] T e388fb22d81e4b08a71710cc04adfcc0 P 906d160bda324389b8c184d19cf04eb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:46:43.163887 12025 raft_consensus.cc:2258] T e388fb22d81e4b08a71710cc04adfcc0 P 906d160bda324389b8c184d19cf04eb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:46:43.164964 12027 ts_tablet_manager.cc:1905] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:46:43.165414 12025 ts_tablet_manager.cc:1905] T e388fb22d81e4b08a71710cc04adfcc0 P 906d160bda324389b8c184d19cf04eb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:46:43.168210 12027 ts_tablet_manager.cc:1918] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:46:43.168324 12027 log.cc:1196] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-0/wal/wals/e388fb22d81e4b08a71710cc04adfcc0
I20260810 01:46:43.168568 12025 ts_tablet_manager.cc:1918] T e388fb22d81e4b08a71710cc04adfcc0 P 906d160bda324389b8c184d19cf04eb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:46:43.168689 12025 log.cc:1196] T e388fb22d81e4b08a71710cc04adfcc0 P 906d160bda324389b8c184d19cf04eb4: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-1/wal/wals/e388fb22d81e4b08a71710cc04adfcc0
I20260810 01:46:43.168707 12027 ts_tablet_manager.cc:1939] T e388fb22d81e4b08a71710cc04adfcc0 P 7c410c8eae2f4ca2a445b4038a2da1f8: Deleting consensus metadata
I20260810 01:46:43.169031 12025 ts_tablet_manager.cc:1939] T e388fb22d81e4b08a71710cc04adfcc0 P 906d160bda324389b8c184d19cf04eb4: Deleting consensus metadata
I20260810 01:46:43.170346 11640 tablet_service.cc:1467] Processing CreateTablet for tablet 2e4f1050f667463c9f1386e0a8b1a6a1 (DEFAULT_TABLE table=test-table [id=1c869d3caefa494c8ac3fec9d5629e9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:46:43.170539 11433 tablet_service.cc:1467] Processing CreateTablet for tablet 2e4f1050f667463c9f1386e0a8b1a6a1 (DEFAULT_TABLE table=test-table [id=1c869d3caefa494c8ac3fec9d5629e9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:46:43.170648 11640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e4f1050f667463c9f1386e0a8b1a6a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:43.170789 11433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e4f1050f667463c9f1386e0a8b1a6a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:43.172344 11854 tablet_service.cc:1467] Processing CreateTablet for tablet 2e4f1050f667463c9f1386e0a8b1a6a1 (DEFAULT_TABLE table=test-table [id=1c869d3caefa494c8ac3fec9d5629e9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:46:43.172647 12026 ts_tablet_manager.cc:1905] T e388fb22d81e4b08a71710cc04adfcc0 P 208e6a0e5f224cf9ac3776e3fa28819b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:46:43.173058 11995 tablet_bootstrap.cc:492] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 906d160bda324389b8c184d19cf04eb4: Bootstrap starting.
I20260810 01:46:43.173506 11196 catalog_manager.cc:4841] TS 906d160bda324389b8c184d19cf04eb4 (127.9.77.130:45649): tablet e388fb22d81e4b08a71710cc04adfcc0 (table test-table [id=c27f33b65eb14c4a828d7ea72a695767]) successfully deleted
I20260810 01:46:43.172673 11854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e4f1050f667463c9f1386e0a8b1a6a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:43.174111 11995 tablet_bootstrap.cc:654] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 906d160bda324389b8c184d19cf04eb4: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:43.174917 11994 tablet_bootstrap.cc:492] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8: Bootstrap starting.
I20260810 01:46:43.175393 11995 tablet_bootstrap.cc:492] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 906d160bda324389b8c184d19cf04eb4: No bootstrap required, opened a new log
I20260810 01:46:43.175475 11995 ts_tablet_manager.cc:1397] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 906d160bda324389b8c184d19cf04eb4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 01:46:43.175931 11994 tablet_bootstrap.cc:654] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:43.176328 11995 raft_consensus.cc:348] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 906d160bda324389b8c184d19cf04eb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } }
I20260810 01:46:43.176466 11995 raft_consensus.cc:374] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 906d160bda324389b8c184d19cf04eb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:43.176507 11995 raft_consensus.cc:729] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 906d160bda324389b8c184d19cf04eb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 906d160bda324389b8c184d19cf04eb4, State: Initialized, Role: FOLLOWER
I20260810 01:46:43.176678 11995 consensus_queue.cc:260] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 906d160bda324389b8c184d19cf04eb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } }
I20260810 01:46:43.176777 11993 tablet_bootstrap.cc:492] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 208e6a0e5f224cf9ac3776e3fa28819b: Bootstrap starting.
I20260810 01:46:43.177591 11995 ts_tablet_manager.cc:1428] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 906d160bda324389b8c184d19cf04eb4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 01:46:43.177820 11196 catalog_manager.cc:4841] TS 7c410c8eae2f4ca2a445b4038a2da1f8 (127.9.77.129:36033): tablet e388fb22d81e4b08a71710cc04adfcc0 (table test-table [id=c27f33b65eb14c4a828d7ea72a695767]) successfully deleted
I20260810 01:46:43.177855 11993 tablet_bootstrap.cc:654] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 208e6a0e5f224cf9ac3776e3fa28819b: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:43.179312 11993 tablet_bootstrap.cc:492] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 208e6a0e5f224cf9ac3776e3fa28819b: No bootstrap required, opened a new log
I20260810 01:46:43.179430 11993 ts_tablet_manager.cc:1397] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 208e6a0e5f224cf9ac3776e3fa28819b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 01:46:43.179880 11993 raft_consensus.cc:348] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 208e6a0e5f224cf9ac3776e3fa28819b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } }
I20260810 01:46:43.180011 11993 raft_consensus.cc:374] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 208e6a0e5f224cf9ac3776e3fa28819b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:43.180069 11993 raft_consensus.cc:729] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 208e6a0e5f224cf9ac3776e3fa28819b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 208e6a0e5f224cf9ac3776e3fa28819b, State: Initialized, Role: FOLLOWER
I20260810 01:46:43.180214 11993 consensus_queue.cc:260] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 208e6a0e5f224cf9ac3776e3fa28819b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } }
I20260810 01:46:43.180588 11993 ts_tablet_manager.cc:1428] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 208e6a0e5f224cf9ac3776e3fa28819b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 01:46:43.182302 12026 ts_tablet_manager.cc:1918] T e388fb22d81e4b08a71710cc04adfcc0 P 208e6a0e5f224cf9ac3776e3fa28819b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:46:43.182416 12026 log.cc:1196] T e388fb22d81e4b08a71710cc04adfcc0 P 208e6a0e5f224cf9ac3776e3fa28819b: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-2/wal/wals/e388fb22d81e4b08a71710cc04adfcc0
I20260810 01:46:43.183017 11994 tablet_bootstrap.cc:492] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8: No bootstrap required, opened a new log
I20260810 01:46:43.183112 11994 ts_tablet_manager.cc:1397] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20260810 01:46:43.183643 11994 raft_consensus.cc:348] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } }
I20260810 01:46:43.183772 11994 raft_consensus.cc:374] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:43.183807 11994 raft_consensus.cc:729] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c410c8eae2f4ca2a445b4038a2da1f8, State: Initialized, Role: FOLLOWER
I20260810 01:46:43.183948 11994 consensus_queue.cc:260] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } }
I20260810 01:46:43.184226 11994 ts_tablet_manager.cc:1428] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260810 01:46:43.184370 12026 ts_tablet_manager.cc:1939] T e388fb22d81e4b08a71710cc04adfcc0 P 208e6a0e5f224cf9ac3776e3fa28819b: Deleting consensus metadata
I20260810 01:46:43.185983 11196 catalog_manager.cc:4841] TS 208e6a0e5f224cf9ac3776e3fa28819b (127.9.77.131:33947): tablet e388fb22d81e4b08a71710cc04adfcc0 (table test-table [id=c27f33b65eb14c4a828d7ea72a695767]) successfully deleted
I20260810 01:46:43.578264 12003 raft_consensus.cc:482] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:46:43.578565 12003 raft_consensus.cc:504] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } }
I20260810 01:46:43.579164 12003 leader_election.cc:288] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 906d160bda324389b8c184d19cf04eb4 (127.9.77.130:45649), 208e6a0e5f224cf9ac3776e3fa28819b (127.9.77.131:33947)
I20260810 01:46:43.580001 11671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4f1050f667463c9f1386e0a8b1a6a1" candidate_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "906d160bda324389b8c184d19cf04eb4" is_pre_election: true
I20260810 01:46:43.580113 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4f1050f667463c9f1386e0a8b1a6a1" candidate_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" is_pre_election: true
I20260810 01:46:43.580262 11671 raft_consensus.cc:2454] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 906d160bda324389b8c184d19cf04eb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c410c8eae2f4ca2a445b4038a2da1f8 in term 0.
I20260810 01:46:43.580327 11887 raft_consensus.cc:2454] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 208e6a0e5f224cf9ac3776e3fa28819b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c410c8eae2f4ca2a445b4038a2da1f8 in term 0.
I20260810 01:46:43.580662 11363 leader_election.cc:302] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c410c8eae2f4ca2a445b4038a2da1f8, 906d160bda324389b8c184d19cf04eb4; no voters:
I20260810 01:46:43.580899 12003 raft_consensus.cc:2781] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:46:43.580994 12003 raft_consensus.cc:482] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:46:43.581050 12003 raft_consensus.cc:3037] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:43.582450 12003 raft_consensus.cc:504] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } }
I20260810 01:46:43.582842 12003 leader_election.cc:288] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8 [CANDIDATE]: Term 1 election: Requested vote from peers 906d160bda324389b8c184d19cf04eb4 (127.9.77.130:45649), 208e6a0e5f224cf9ac3776e3fa28819b (127.9.77.131:33947)
I20260810 01:46:43.583586 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4f1050f667463c9f1386e0a8b1a6a1" candidate_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "208e6a0e5f224cf9ac3776e3fa28819b"
I20260810 01:46:43.583586 11671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4f1050f667463c9f1386e0a8b1a6a1" candidate_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "906d160bda324389b8c184d19cf04eb4"
I20260810 01:46:43.583736 11671 raft_consensus.cc:3037] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 906d160bda324389b8c184d19cf04eb4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:43.583736 11887 raft_consensus.cc:3037] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 208e6a0e5f224cf9ac3776e3fa28819b [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:43.585170 11887 raft_consensus.cc:2454] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 208e6a0e5f224cf9ac3776e3fa28819b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c410c8eae2f4ca2a445b4038a2da1f8 in term 1.
I20260810 01:46:43.585168 11671 raft_consensus.cc:2454] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 906d160bda324389b8c184d19cf04eb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c410c8eae2f4ca2a445b4038a2da1f8 in term 1.
I20260810 01:46:43.585513 11363 leader_election.cc:302] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c410c8eae2f4ca2a445b4038a2da1f8, 906d160bda324389b8c184d19cf04eb4; no voters:
I20260810 01:46:43.585770 12003 raft_consensus.cc:2781] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:46:43.585913 12003 raft_consensus.cc:686] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 1 LEADER]: Becoming Leader. State: Replica: 7c410c8eae2f4ca2a445b4038a2da1f8, State: Running, Role: LEADER
I20260810 01:46:43.586102 12003 consensus_queue.cc:237] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } }
I20260810 01:46:43.588313 11219 catalog_manager.cc:5495] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c410c8eae2f4ca2a445b4038a2da1f8 (127.9.77.129). New cstate: current_term: 1 leader_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } health_report { overall_health: UNKNOWN } } }
Aug 10 01:46:43 dist-test-slave-kvfs krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326401, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:43.668902 12047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:43.658790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45779 (local address 127.0.0.1:59506)
0810 01:46:43.659381 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:43.659397 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:43.659564 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:43.659947 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:43.659951 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:43.660531 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:46:43.660659 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:43.660665 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:43.661572 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:43.661574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:43.662177 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:43.662185 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:43.662676 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:43.662678 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:43.662778 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:43.663474 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:46:43.663487 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:46:43.665245 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:46:43.667141 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:43.667151 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:43.667157 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:43.667460 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:43.667743 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:43.667745 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:43.667746 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:43.667840 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:43.668291 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:46:43.668297 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:46:43.668419 (+ 122us) client_negotiation.cc:769] Sending connection context
0810 01:46:43.668533 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 01:46:43.668676 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":29,"thread_start_us":166,"threads_started":1}
I20260810 01:46:43.668919 12048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:43.658854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59506 (local address 127.9.77.190:45779)
0810 01:46:43.659244 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:43.659258 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:43.659511 (+ 253us) server_negotiation.cc:407] Connection header received
0810 01:46:43.659675 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:43.659679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:43.659737 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:43.659877 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:43.660811 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:43.661466 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:43.662290 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:43.662573 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:43.662660 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:43.665386 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:43.665389 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:43.665392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:43.665414 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:43.667022 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:43.667568 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:43.667571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:43.667572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:43.667654 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:43.667941 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:43.667943 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:43.667945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:43.668186 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:43.668268 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:43.668585 (+ 317us) server_negotiation.cc:299] Negotiation successful
0810 01:46:43.668733 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":269,"thread_start_us":128,"threads_started":1}
I20260810 01:46:43.674036 11219 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:59506:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 01:46:43.675000 11219 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:46:43.680819 11433 tablet_service.cc:1467] Processing CreateTablet for tablet f01375e577d640e9b0f2f34fbb7a7331 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=af4a5524d34044ce8e0e4d4c9671192d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 01:46:43.680878 11854 tablet_service.cc:1467] Processing CreateTablet for tablet f01375e577d640e9b0f2f34fbb7a7331 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=af4a5524d34044ce8e0e4d4c9671192d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 01:46:43.680864 11640 tablet_service.cc:1467] Processing CreateTablet for tablet f01375e577d640e9b0f2f34fbb7a7331 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=af4a5524d34044ce8e0e4d4c9671192d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 01:46:43.681365 11433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f01375e577d640e9b0f2f34fbb7a7331. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:43.681366 11854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f01375e577d640e9b0f2f34fbb7a7331. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:43.681398 11640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f01375e577d640e9b0f2f34fbb7a7331. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:43.685431 12050 tablet_bootstrap.cc:492] T f01375e577d640e9b0f2f34fbb7a7331 P 906d160bda324389b8c184d19cf04eb4: Bootstrap starting.
I20260810 01:46:43.686713 12051 tablet_bootstrap.cc:492] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8: Bootstrap starting.
I20260810 01:46:43.686746 12050 tablet_bootstrap.cc:654] T f01375e577d640e9b0f2f34fbb7a7331 P 906d160bda324389b8c184d19cf04eb4: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:43.687597 12049 tablet_bootstrap.cc:492] T f01375e577d640e9b0f2f34fbb7a7331 P 208e6a0e5f224cf9ac3776e3fa28819b: Bootstrap starting.
I20260810 01:46:43.687672 12051 tablet_bootstrap.cc:654] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:43.688763 12049 tablet_bootstrap.cc:654] T f01375e577d640e9b0f2f34fbb7a7331 P 208e6a0e5f224cf9ac3776e3fa28819b: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:43.689203 12051 tablet_bootstrap.cc:492] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8: No bootstrap required, opened a new log
I20260810 01:46:43.689328 12051 ts_tablet_manager.cc:1397] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 01:46:43.689769 12050 tablet_bootstrap.cc:492] T f01375e577d640e9b0f2f34fbb7a7331 P 906d160bda324389b8c184d19cf04eb4: No bootstrap required, opened a new log
I20260810 01:46:43.689915 12050 ts_tablet_manager.cc:1397] T f01375e577d640e9b0f2f34fbb7a7331 P 906d160bda324389b8c184d19cf04eb4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260810 01:46:43.689865 12051 raft_consensus.cc:348] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } }
I20260810 01:46:43.690039 12051 raft_consensus.cc:374] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:43.690083 12051 raft_consensus.cc:729] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c410c8eae2f4ca2a445b4038a2da1f8, State: Initialized, Role: FOLLOWER
I20260810 01:46:43.690254 12051 consensus_queue.cc:260] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } }
I20260810 01:46:43.690478 12050 raft_consensus.cc:348] T f01375e577d640e9b0f2f34fbb7a7331 P 906d160bda324389b8c184d19cf04eb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } }
I20260810 01:46:43.690600 12050 raft_consensus.cc:374] T f01375e577d640e9b0f2f34fbb7a7331 P 906d160bda324389b8c184d19cf04eb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:43.690632 12050 raft_consensus.cc:729] T f01375e577d640e9b0f2f34fbb7a7331 P 906d160bda324389b8c184d19cf04eb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 906d160bda324389b8c184d19cf04eb4, State: Initialized, Role: FOLLOWER
I20260810 01:46:43.690616 12051 ts_tablet_manager.cc:1428] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 01:46:43.690616 12003 tablet_replica.cc:439] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } } }
I20260810 01:46:43.690810 12003 tablet_replica.cc:442] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 01:46:43.690753 12050 consensus_queue.cc:260] T f01375e577d640e9b0f2f34fbb7a7331 P 906d160bda324389b8c184d19cf04eb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } }
I20260810 01:46:43.691114 12050 ts_tablet_manager.cc:1428] T f01375e577d640e9b0f2f34fbb7a7331 P 906d160bda324389b8c184d19cf04eb4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 01:46:43.691121 12055 tablet_replica.cc:439] T f01375e577d640e9b0f2f34fbb7a7331 P 906d160bda324389b8c184d19cf04eb4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } } }
I20260810 01:46:43.691367 12055 tablet_replica.cc:442] T f01375e577d640e9b0f2f34fbb7a7331 P 906d160bda324389b8c184d19cf04eb4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 01:46:43.693130 12049 tablet_bootstrap.cc:492] T f01375e577d640e9b0f2f34fbb7a7331 P 208e6a0e5f224cf9ac3776e3fa28819b: No bootstrap required, opened a new log
I20260810 01:46:43.693244 12049 ts_tablet_manager.cc:1397] T f01375e577d640e9b0f2f34fbb7a7331 P 208e6a0e5f224cf9ac3776e3fa28819b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260810 01:46:43.693842 12049 raft_consensus.cc:348] T f01375e577d640e9b0f2f34fbb7a7331 P 208e6a0e5f224cf9ac3776e3fa28819b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } }
I20260810 01:46:43.694031 12049 raft_consensus.cc:374] T f01375e577d640e9b0f2f34fbb7a7331 P 208e6a0e5f224cf9ac3776e3fa28819b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:43.694070 12049 raft_consensus.cc:729] T f01375e577d640e9b0f2f34fbb7a7331 P 208e6a0e5f224cf9ac3776e3fa28819b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 208e6a0e5f224cf9ac3776e3fa28819b, State: Initialized, Role: FOLLOWER
I20260810 01:46:43.694219 12049 consensus_queue.cc:260] T f01375e577d640e9b0f2f34fbb7a7331 P 208e6a0e5f224cf9ac3776e3fa28819b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } }
I20260810 01:46:43.694660 12049 ts_tablet_manager.cc:1428] T f01375e577d640e9b0f2f34fbb7a7331 P 208e6a0e5f224cf9ac3776e3fa28819b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 01:46:43.694651 12056 tablet_replica.cc:439] T f01375e577d640e9b0f2f34fbb7a7331 P 208e6a0e5f224cf9ac3776e3fa28819b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } } }
I20260810 01:46:43.694844 12056 tablet_replica.cc:442] T f01375e577d640e9b0f2f34fbb7a7331 P 208e6a0e5f224cf9ac3776e3fa28819b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 01:46:43.844431 12003 raft_consensus.cc:482] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:46:43.844683 12003 raft_consensus.cc:504] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } }
I20260810 01:46:43.845285 12003 leader_election.cc:288] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 906d160bda324389b8c184d19cf04eb4 (127.9.77.130:45649), 208e6a0e5f224cf9ac3776e3fa28819b (127.9.77.131:33947)
I20260810 01:46:43.845918 11671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f01375e577d640e9b0f2f34fbb7a7331" candidate_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "906d160bda324389b8c184d19cf04eb4" is_pre_election: true
I20260810 01:46:43.846153 11671 raft_consensus.cc:2454] T f01375e577d640e9b0f2f34fbb7a7331 P 906d160bda324389b8c184d19cf04eb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c410c8eae2f4ca2a445b4038a2da1f8 in term 0.
I20260810 01:46:43.846093 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f01375e577d640e9b0f2f34fbb7a7331" candidate_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" is_pre_election: true
I20260810 01:46:43.846412 11887 raft_consensus.cc:2454] T f01375e577d640e9b0f2f34fbb7a7331 P 208e6a0e5f224cf9ac3776e3fa28819b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c410c8eae2f4ca2a445b4038a2da1f8 in term 0.
I20260810 01:46:43.846602 11363 leader_election.cc:302] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c410c8eae2f4ca2a445b4038a2da1f8, 906d160bda324389b8c184d19cf04eb4; no voters:
I20260810 01:46:43.846864 12003 raft_consensus.cc:2781] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:46:43.846958 12003 raft_consensus.cc:482] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:46:43.847016 12003 raft_consensus.cc:3037] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:43.848371 12003 raft_consensus.cc:504] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } }
I20260810 01:46:43.848734 12003 leader_election.cc:288] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8 [CANDIDATE]: Term 1 election: Requested vote from peers 906d160bda324389b8c184d19cf04eb4 (127.9.77.130:45649), 208e6a0e5f224cf9ac3776e3fa28819b (127.9.77.131:33947)
I20260810 01:46:43.849263 11887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f01375e577d640e9b0f2f34fbb7a7331" candidate_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "208e6a0e5f224cf9ac3776e3fa28819b"
I20260810 01:46:43.849292 11671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f01375e577d640e9b0f2f34fbb7a7331" candidate_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "906d160bda324389b8c184d19cf04eb4"
I20260810 01:46:43.849417 11887 raft_consensus.cc:3037] T f01375e577d640e9b0f2f34fbb7a7331 P 208e6a0e5f224cf9ac3776e3fa28819b [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:43.849449 11671 raft_consensus.cc:3037] T f01375e577d640e9b0f2f34fbb7a7331 P 906d160bda324389b8c184d19cf04eb4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:43.850554 11671 raft_consensus.cc:2454] T f01375e577d640e9b0f2f34fbb7a7331 P 906d160bda324389b8c184d19cf04eb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c410c8eae2f4ca2a445b4038a2da1f8 in term 1.
I20260810 01:46:43.850585 11887 raft_consensus.cc:2454] T f01375e577d640e9b0f2f34fbb7a7331 P 208e6a0e5f224cf9ac3776e3fa28819b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c410c8eae2f4ca2a445b4038a2da1f8 in term 1.
I20260810 01:46:43.850880 11363 leader_election.cc:302] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c410c8eae2f4ca2a445b4038a2da1f8, 906d160bda324389b8c184d19cf04eb4; no voters:
I20260810 01:46:43.851092 12003 raft_consensus.cc:2781] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:46:43.851244 12003 raft_consensus.cc:686] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 1 LEADER]: Becoming Leader. State: Replica: 7c410c8eae2f4ca2a445b4038a2da1f8, State: Running, Role: LEADER
I20260810 01:46:43.851436 12003 consensus_queue.cc:237] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } }
I20260810 01:46:43.852591 12021 tablet_replica.cc:439] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8: TxnStatusTablet state changed. Reason: New leader 7c410c8eae2f4ca2a445b4038a2da1f8. Latest consensus state: current_term: 1 leader_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } } }
I20260810 01:46:43.852716 12021 tablet_replica.cc:442] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8: This TxnStatusTablet replica's current role is: LEADER
I20260810 01:46:43.853082 12058 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 01:46:43.853322 12058 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 01:46:43.853847 11218 catalog_manager.cc:5495] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c410c8eae2f4ca2a445b4038a2da1f8 (127.9.77.129). New cstate: current_term: 1 leader_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } health_report { overall_health: UNKNOWN } } }
I20260810 01:46:43.881831 12061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:43.877976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53680 (local address 127.9.77.129:36033)
0810 01:46:43.878386 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:43.878393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:43.878406 (+ 13us) server_negotiation.cc:407] Connection header received
0810 01:46:43.878463 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:43.878465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:43.878515 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:43.878626 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:46:43.878947 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:43.879620 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:43.880594 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:43.880849 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:43.880963 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:43.881275 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:46:43.881374 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:46:43.881457 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:43.881586 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 01:46:43.881643 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":157,"threads_started":1}
I20260810 01:46:43.881829 12047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:43.877742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:36033 (local address 127.0.0.1:53680)
0810 01:46:43.877963 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:43.877982 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:43.878125 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:43.878657 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:43.878660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:43.878669 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 01:46:43.878817 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:43.878823 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:43.879744 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:43.879750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:43.880464 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:43.880474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:43.880975 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:43.880978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:43.881075 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:43.881125 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 01:46:43.881462 (+ 337us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 01:46:43.881483 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 01:46:43.881572 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 01:46:43.881679 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260810 01:46:43.885725 11887 raft_consensus.cc:1261] T f01375e577d640e9b0f2f34fbb7a7331 P 208e6a0e5f224cf9ac3776e3fa28819b [term 1 FOLLOWER]: Refusing update from remote peer 7c410c8eae2f4ca2a445b4038a2da1f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:46:43.885802 11671 raft_consensus.cc:1261] T f01375e577d640e9b0f2f34fbb7a7331 P 906d160bda324389b8c184d19cf04eb4 [term 1 FOLLOWER]: Refusing update from remote peer 7c410c8eae2f4ca2a445b4038a2da1f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:46:43.886332 12021 consensus_queue.cc:1035] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:46:43.886489 12003 consensus_queue.cc:1035] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:46:43.888345 12056 tablet_replica.cc:439] T f01375e577d640e9b0f2f34fbb7a7331 P 208e6a0e5f224cf9ac3776e3fa28819b: TxnStatusTablet state changed. Reason: New leader 7c410c8eae2f4ca2a445b4038a2da1f8. Latest consensus state: current_term: 1 leader_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } } }
I20260810 01:46:43.888381 12055 tablet_replica.cc:439] T f01375e577d640e9b0f2f34fbb7a7331 P 906d160bda324389b8c184d19cf04eb4: TxnStatusTablet state changed. Reason: New leader 7c410c8eae2f4ca2a445b4038a2da1f8. Latest consensus state: current_term: 1 leader_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } } }
I20260810 01:46:43.888479 12056 tablet_replica.cc:442] T f01375e577d640e9b0f2f34fbb7a7331 P 208e6a0e5f224cf9ac3776e3fa28819b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 01:46:43.888504 12055 tablet_replica.cc:442] T f01375e577d640e9b0f2f34fbb7a7331 P 906d160bda324389b8c184d19cf04eb4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 01:46:43.891287 12003 tablet_replica.cc:439] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } } }
I20260810 01:46:43.891423 12003 tablet_replica.cc:442] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8: This TxnStatusTablet replica's current role is: LEADER
I20260810 01:46:43.891855 12055 tablet_replica.cc:439] T f01375e577d640e9b0f2f34fbb7a7331 P 906d160bda324389b8c184d19cf04eb4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } } }
I20260810 01:46:43.891973 12055 tablet_replica.cc:442] T f01375e577d640e9b0f2f34fbb7a7331 P 906d160bda324389b8c184d19cf04eb4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 01:46:43.897845 12065 mvcc.cc:205] Tried to move back new op lower bound from 7316792950310699008 to 7316792950177234944. Current Snapshot: MvccSnapshot[applied={T|T < 7316792950310699008}]
I20260810 01:46:43.898586 12021 tablet_replica.cc:439] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } } }
I20260810 01:46:43.898729 12021 tablet_replica.cc:442] T f01375e577d640e9b0f2f34fbb7a7331 P 7c410c8eae2f4ca2a445b4038a2da1f8: This TxnStatusTablet replica's current role is: LEADER
I20260810 01:46:43.900090 12056 tablet_replica.cc:439] T f01375e577d640e9b0f2f34fbb7a7331 P 208e6a0e5f224cf9ac3776e3fa28819b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c410c8eae2f4ca2a445b4038a2da1f8" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 36033 } } peers { permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 } } peers { permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 } } }
I20260810 01:46:43.900234 12056 tablet_replica.cc:442] T f01375e577d640e9b0f2f34fbb7a7331 P 208e6a0e5f224cf9ac3776e3fa28819b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 01:46:43.912676 12061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:43.908726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:38827 (local address 127.9.77.129:36033)
0810 01:46:43.908886 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:43.908890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:43.909131 (+ 241us) server_negotiation.cc:407] Connection header received
0810 01:46:43.909254 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:43.909257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:43.909309 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:43.909386 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:46:43.909649 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:43.910490 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:43.911374 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:43.911726 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:43.911830 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:43.912172 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:46:43.912268 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:46:43.912344 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:43.912431 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 01:46:43.912485 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":50}
I20260810 01:46:43.912676 12077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:43.908601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:36033 (local address 127.9.77.129:38827)
0810 01:46:43.909030 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:43.909042 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:43.909156 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:43.909404 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:43.909407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:43.909413 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 01:46:43.909552 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:43.909556 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:43.910605 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:43.910607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:43.911259 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:43.911267 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:43.911863 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:43.911866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:43.911930 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:43.911985 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 01:46:43.912355 (+ 370us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 01:46:43.912360 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 01:46:43.912414 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 01:46:43.912498 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":20,"thread_start_us":138,"threads_started":1}
I20260810 01:46:43.923791 11887 raft_consensus.cc:1261] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 208e6a0e5f224cf9ac3776e3fa28819b [term 1 FOLLOWER]: Refusing update from remote peer 7c410c8eae2f4ca2a445b4038a2da1f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:46:43.924305 11671 raft_consensus.cc:1261] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 906d160bda324389b8c184d19cf04eb4 [term 1 FOLLOWER]: Refusing update from remote peer 7c410c8eae2f4ca2a445b4038a2da1f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:46:43.924455 12021 consensus_queue.cc:1035] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "208e6a0e5f224cf9ac3776e3fa28819b" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 33947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:46:43.924801 12021 consensus_queue.cc:1035] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "906d160bda324389b8c184d19cf04eb4" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:46:43.972775 11219 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:46:43.972991 11219 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:46:43.975548 11219 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4252c02364924a12a3d85ccadf11e1de: Sending DeleteTablet for 3 replicas of tablet 2e4f1050f667463c9f1386e0a8b1a6a1
I20260810 01:46:43.976230 11433 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e4f1050f667463c9f1386e0a8b1a6a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:46:43 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:53662
I20260810 01:46:43.976728 12083 tablet_replica.cc:330] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8: stopping tablet replica
I20260810 01:46:43.976771 11854 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e4f1050f667463c9f1386e0a8b1a6a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:46:43 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:51760
I20260810 01:46:43.976809 11640 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e4f1050f667463c9f1386e0a8b1a6a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:46:43 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:60666
I20260810 01:46:43.977056 12083 raft_consensus.cc:2229] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 1 LEADER]: Raft consensus shutting down.
I20260810 01:46:43.977290 12083 raft_consensus.cc:2258] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:46:43.978024 12083 ts_tablet_manager.cc:1905] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:46:43.978580 12084 tablet_replica.cc:330] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 906d160bda324389b8c184d19cf04eb4: stopping tablet replica
I20260810 01:46:43.978747 12084 raft_consensus.cc:2229] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 906d160bda324389b8c184d19cf04eb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:46:43.978905 12084 raft_consensus.cc:2258] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 906d160bda324389b8c184d19cf04eb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:46:43.979220 12085 tablet_replica.cc:330] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 208e6a0e5f224cf9ac3776e3fa28819b: stopping tablet replica
I20260810 01:46:43.979398 12085 raft_consensus.cc:2229] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 208e6a0e5f224cf9ac3776e3fa28819b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:46:43.979560 12085 raft_consensus.cc:2258] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 208e6a0e5f224cf9ac3776e3fa28819b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:46:43.979704 12084 ts_tablet_manager.cc:1905] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 906d160bda324389b8c184d19cf04eb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:46:43.980415 12085 ts_tablet_manager.cc:1905] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 208e6a0e5f224cf9ac3776e3fa28819b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:46:43.983973 12085 ts_tablet_manager.cc:1918] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 208e6a0e5f224cf9ac3776e3fa28819b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 01:46:43.984097 12085 log.cc:1196] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 208e6a0e5f224cf9ac3776e3fa28819b: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-2/wal/wals/2e4f1050f667463c9f1386e0a8b1a6a1
I20260810 01:46:43.984448 12085 ts_tablet_manager.cc:1939] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 208e6a0e5f224cf9ac3776e3fa28819b: Deleting consensus metadata
I20260810 01:46:43.984546 12084 ts_tablet_manager.cc:1918] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 906d160bda324389b8c184d19cf04eb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 01:46:43.984643 12084 log.cc:1196] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 906d160bda324389b8c184d19cf04eb4: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-1/wal/wals/2e4f1050f667463c9f1386e0a8b1a6a1
I20260810 01:46:43.984958 12084 ts_tablet_manager.cc:1939] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 906d160bda324389b8c184d19cf04eb4: Deleting consensus metadata
I20260810 01:46:43.985476 11196 catalog_manager.cc:4841] TS 208e6a0e5f224cf9ac3776e3fa28819b (127.9.77.131:33947): tablet 2e4f1050f667463c9f1386e0a8b1a6a1 (table test-table [id=1c869d3caefa494c8ac3fec9d5629e9e]) successfully deleted
I20260810 01:46:43.985773 11196 catalog_manager.cc:4841] TS 906d160bda324389b8c184d19cf04eb4 (127.9.77.130:45649): tablet 2e4f1050f667463c9f1386e0a8b1a6a1 (table test-table [id=1c869d3caefa494c8ac3fec9d5629e9e]) successfully deleted
I20260810 01:46:43.987602 12083 ts_tablet_manager.cc:1918] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 01:46:43.987696 12083 log.cc:1196] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786326392877349-9526-0/minicluster-data/ts-0/wal/wals/2e4f1050f667463c9f1386e0a8b1a6a1
I20260810 01:46:43.987967 12083 ts_tablet_manager.cc:1939] T 2e4f1050f667463c9f1386e0a8b1a6a1 P 7c410c8eae2f4ca2a445b4038a2da1f8: Deleting consensus metadata
I20260810 01:46:43.988863 11196 catalog_manager.cc:4841] TS 7c410c8eae2f4ca2a445b4038a2da1f8 (127.9.77.129:36033): tablet 2e4f1050f667463c9f1386e0a8b1a6a1 (table test-table [id=1c869d3caefa494c8ac3fec9d5629e9e]) successfully deleted
Aug 10 01:46:43 dist-test-slave-kvfs krb5kdc[11149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326402, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.77.129@KRBTEST.COM
I20260810 01:46:43.996248 12061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:43.978693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53688 (local address 127.9.77.129:36033)
0810 01:46:43.978845 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:43.978850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:43.979614 (+ 764us) server_negotiation.cc:407] Connection header received
0810 01:46:43.979955 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:43.979959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:43.980067 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:43.980130 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:43.981300 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:43.982104 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:43.983396 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:43.983748 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:43.983861 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:43.991929 (+ 8068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:43.991946 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:43.991971 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:43.992008 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:43.993996 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:43.994578 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:43.994585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:43.994586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:43.994677 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:43.995004 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:43.995007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:43.995008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:43.995316 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:43.995528 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:43.995867 (+ 339us) server_negotiation.cc:299] Negotiation successful
0810 01:46:43.995990 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260810 01:46:43.996939 11523 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53688
I20260810 01:46:44.007540 12048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:43.999220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59508 (local address 127.9.77.190:45779)
0810 01:46:43.999413 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:43.999420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:43.999436 (+ 16us) server_negotiation.cc:407] Connection header received
0810 01:46:43.999594 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:43.999602 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:43.999657 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:43.999753 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:44.000552 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:44.001155 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:44.001998 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:44.002262 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:44.002389 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:44.003867 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:44.003873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:44.003881 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:44.003909 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:44.005640 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:44.006110 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:44.006114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:44.006116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:44.006210 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:44.006504 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:44.006508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:44.006510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:44.006728 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:44.006890 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:44.007139 (+ 249us) server_negotiation.cc:299] Negotiation successful
0810 01:46:44.007327 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260810 01:46:44.008087 11219 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59508
I20260810 01:46:44.009577 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 11337
I20260810 01:46:44.029839 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 11544
I20260810 01:46:44.047096 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 11755
I20260810 01:46:44.058951 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 11167
2026-08-10T01:46:44Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3870 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:46:44 dist-test-slave-kvfs krb5kdc[12097](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:46:44 dist-test-slave-kvfs krb5kdc[12097](info): set up 2 sockets
Aug 10 01:46:44 dist-test-slave-kvfs krb5kdc[12097](info): commencing operation
krb5kdc: starting...
W20260810 01:46:45.367786 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.271s user 0.001s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:46:45 dist-test-slave-kvfs krb5kdc[12097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:46:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:46:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:46:45.519927 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:46807
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:42529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:46807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:46:45.654837 12112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:46:45.655256 12112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:46:45.655335 12112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:46:45.659741 12112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:46:45.659855 12112 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 01:46:45.659886 12112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:46:45.659929 12112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:46:45.659988 12112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:46:45.660045 12112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:46:45.665987 12112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:42529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:46807
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:46807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:46:45.667533 12112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:46:45.668905 12112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:46:45.679605 12124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:45.679605 12122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:45.679682 12127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:45.679939 12112 server_base.cc:1034] running on GCE node
I20260810 01:46:45.680754 12112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:46:45.682255 12112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:46:45.683473 12112 hybrid_clock.cc:648] HybridClock initialized: now 1786326405683450 us; error 36 us; skew 500 ppm
Aug 10 01:46:45 dist-test-slave-kvfs krb5kdc[12097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326405, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:46:45.686857 12112 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:46:45.687994 12112 webserver.cc:460] Webserver started at http://127.9.77.190:45995/ using document root <none> and password file <none>
I20260810 01:46:45.688769 12112 fs_manager.cc:359] Metadata directory not provided
I20260810 01:46:45.688903 12112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:46:45.689190 12112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:46:45.691780 12112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "ad82f728b5a5494a9f9deaf9c241c54f"
format_stamp: "Formatted at 2026-08-10 01:46:45 on dist-test-slave-kvfs"
server_key: "c2626f448c7222f7f512125d83532682"
server_key_iv: "51cfa03df6348c28c77fbe20c77a6164"
server_key_version: "encryptionkey@0"
I20260810 01:46:45.692534 12112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "ad82f728b5a5494a9f9deaf9c241c54f"
format_stamp: "Formatted at 2026-08-10 01:46:45 on dist-test-slave-kvfs"
server_key: "c2626f448c7222f7f512125d83532682"
server_key_iv: "51cfa03df6348c28c77fbe20c77a6164"
server_key_version: "encryptionkey@0"
I20260810 01:46:45.697186 12112 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 01:46:45.700361 12137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:45.702054 12112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 01:46:45.702245 12112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "ad82f728b5a5494a9f9deaf9c241c54f"
format_stamp: "Formatted at 2026-08-10 01:46:45 on dist-test-slave-kvfs"
server_key: "c2626f448c7222f7f512125d83532682"
server_key_iv: "51cfa03df6348c28c77fbe20c77a6164"
server_key_version: "encryptionkey@0"
I20260810 01:46:45.702396 12112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:46:45.720664 12112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:46:45.724803 12112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:46:45.725087 12112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:46:45.735405 12112 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:46807
I20260810 01:46:45.735441 12234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:46807 every 8 connection(s)
I20260810 01:46:45.736835 12112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:46:45.739883 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 12112
I20260810 01:46:45.740056 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:46:45.740442 9526 external_mini_cluster.cc:1428] Setting key e848456ea65808dddf383877a9790ca8
I20260810 01:46:45.741179 12235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:45.749220 12235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad82f728b5a5494a9f9deaf9c241c54f: Bootstrap starting.
Aug 10 01:46:45 dist-test-slave-kvfs krb5kdc[12097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:45.752912 12235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad82f728b5a5494a9f9deaf9c241c54f: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:45.754272 12235 log.cc:824] T 00000000000000000000000000000000 P ad82f728b5a5494a9f9deaf9c241c54f: Log is configured to *not* fsync() on all Append() calls
I20260810 01:46:45.757136 12235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad82f728b5a5494a9f9deaf9c241c54f: No bootstrap required, opened a new log
I20260810 01:46:45.758018 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:45.742453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41252 (local address 127.9.77.190:46807)
0810 01:46:45.743054 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:45.743075 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:45.743122 (+ 47us) server_negotiation.cc:407] Connection header received
0810 01:46:45.743935 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:45.743955 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:45.744451 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:45.744852 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:45.745798 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:45.746925 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:45.747865 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:45.748486 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:45.748678 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:45.751962 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:45.751969 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:45.751986 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:45.752072 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:45.754184 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:45.754760 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:45.754765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:45.754772 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:45.754926 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:45.755247 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:45.755250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:45.755252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:45.755771 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:45.756019 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:45.756407 (+ 388us) server_negotiation.cc:299] Negotiation successful
0810 01:46:45.757001 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":189,"threads_started":1}
I20260810 01:46:45.761507 12235 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad82f728b5a5494a9f9deaf9c241c54f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad82f728b5a5494a9f9deaf9c241c54f" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46807 } }
I20260810 01:46:45.761859 12235 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad82f728b5a5494a9f9deaf9c241c54f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:45.761955 12235 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad82f728b5a5494a9f9deaf9c241c54f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad82f728b5a5494a9f9deaf9c241c54f, State: Initialized, Role: FOLLOWER
I20260810 01:46:45.762508 12235 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad82f728b5a5494a9f9deaf9c241c54f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad82f728b5a5494a9f9deaf9c241c54f" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46807 } }
I20260810 01:46:45.762718 12235 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad82f728b5a5494a9f9deaf9c241c54f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:46:45.762816 12235 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad82f728b5a5494a9f9deaf9c241c54f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:46:45.762972 12235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad82f728b5a5494a9f9deaf9c241c54f [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:45.764317 12235 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad82f728b5a5494a9f9deaf9c241c54f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad82f728b5a5494a9f9deaf9c241c54f" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46807 } }
I20260810 01:46:45.764773 12235 leader_election.cc:302] T 00000000000000000000000000000000 P ad82f728b5a5494a9f9deaf9c241c54f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad82f728b5a5494a9f9deaf9c241c54f; no voters:
I20260810 01:46:45.765272 12235 leader_election.cc:288] T 00000000000000000000000000000000 P ad82f728b5a5494a9f9deaf9c241c54f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:46:45.765450 12242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad82f728b5a5494a9f9deaf9c241c54f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:46:45.765727 12242 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad82f728b5a5494a9f9deaf9c241c54f [term 1 LEADER]: Becoming Leader. State: Replica: ad82f728b5a5494a9f9deaf9c241c54f, State: Running, Role: LEADER
I20260810 01:46:45.766136 12242 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad82f728b5a5494a9f9deaf9c241c54f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad82f728b5a5494a9f9deaf9c241c54f" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46807 } }
I20260810 01:46:45.766983 12235 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad82f728b5a5494a9f9deaf9c241c54f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:46:45.767604 12242 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad82f728b5a5494a9f9deaf9c241c54f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad82f728b5a5494a9f9deaf9c241c54f. Latest consensus state: current_term: 1 leader_uuid: "ad82f728b5a5494a9f9deaf9c241c54f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad82f728b5a5494a9f9deaf9c241c54f" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46807 } } }
I20260810 01:46:45.767738 12242 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad82f728b5a5494a9f9deaf9c241c54f [sys.catalog]: This master's current role is: LEADER
I20260810 01:46:45.768250 12244 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad82f728b5a5494a9f9deaf9c241c54f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad82f728b5a5494a9f9deaf9c241c54f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad82f728b5a5494a9f9deaf9c241c54f" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46807 } } }
I20260810 01:46:45.768435 12244 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad82f728b5a5494a9f9deaf9c241c54f [sys.catalog]: This master's current role is: LEADER
I20260810 01:46:45.768357 12246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:46:45.772887 12246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:46:45.780170 12246 catalog_manager.cc:1347] Generated new cluster ID: b3fc081b6b7049918e20c67276d448ba
I20260810 01:46:45.780285 12246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:46:45.788055 12246 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:46:45.789294 12246 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:46:45.798004 12246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad82f728b5a5494a9f9deaf9c241c54f: Generated new TSK 0
I20260810 01:46:45.799132 12246 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.77.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
I20260810 01:46:45.845016 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:0
--local_ip_for_outbound_sockets=127.9.77.129
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:46807
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:42529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:46:45.991532 12275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:46:45.991935 12275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:46:45.991991 12275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:46:45.996502 12275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:46:45.996585 12275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:46:45.996631 12275 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 01:46:45.996646 12275 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 01:46:45.996721 12275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:46:46.002748 12275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:42529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.77.190:46807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:46:46.004372 12275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:46:46.005890 12275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:46:46.019057 12287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:46.019058 12285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:46.019057 12284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:46.019340 12275 server_base.cc:1034] running on GCE node
I20260810 01:46:46.020084 12275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:46:46.020992 12275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:46:46.022212 12275 hybrid_clock.cc:648] HybridClock initialized: now 1786326406022191 us; error 57 us; skew 500 ppm
Aug 10 01:46:46 dist-test-slave-kvfs krb5kdc[12097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326406, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:46:46.026072 12275 init.cc:377] Logged in from keytab as kudu/127.9.77.129@KRBTEST.COM (short username kudu)
I20260810 01:46:46.027529 12275 webserver.cc:460] Webserver started at http://127.9.77.129:45945/ using document root <none> and password file <none>
I20260810 01:46:46.028383 12275 fs_manager.cc:359] Metadata directory not provided
I20260810 01:46:46.028465 12275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:46:46.028695 12275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:46:46.031327 12275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-0/data/instance:
uuid: "40a6de2a6f85463998b46a214b210bb2"
format_stamp: "Formatted at 2026-08-10 01:46:46 on dist-test-slave-kvfs"
server_key: "f5aa3c658b7dda1fa6bc8be35438ed51"
server_key_iv: "03b46ba266d9e69e6a62d1b0264902f8"
server_key_version: "encryptionkey@0"
I20260810 01:46:46.032075 12275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance:
uuid: "40a6de2a6f85463998b46a214b210bb2"
format_stamp: "Formatted at 2026-08-10 01:46:46 on dist-test-slave-kvfs"
server_key: "f5aa3c658b7dda1fa6bc8be35438ed51"
server_key_iv: "03b46ba266d9e69e6a62d1b0264902f8"
server_key_version: "encryptionkey@0"
I20260810 01:46:46.037212 12275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 01:46:46.040853 12296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:46.042639 12275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:46:46.042834 12275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "40a6de2a6f85463998b46a214b210bb2"
format_stamp: "Formatted at 2026-08-10 01:46:46 on dist-test-slave-kvfs"
server_key: "f5aa3c658b7dda1fa6bc8be35438ed51"
server_key_iv: "03b46ba266d9e69e6a62d1b0264902f8"
server_key_version: "encryptionkey@0"
I20260810 01:46:46.042989 12275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:46:46.065145 12275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:46:46.069571 12275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:46:46.069823 12275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:46:46.071767 12275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:46:46.071826 12275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:46.071900 12275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:46:46.071940 12275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 01:46:46 dist-test-slave-kvfs krb5kdc[12097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326406, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:46.085515 12275 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:39511
I20260810 01:46:46.085551 12466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:39511 every 8 connection(s)
I20260810 01:46:46.086863 12275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:46:46.088626 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:46.072967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:42859 (local address 127.9.77.190:46807)
0810 01:46:46.073163 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:46.073170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:46.074257 (+ 1087us) server_negotiation.cc:407] Connection header received
0810 01:46:46.075621 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:46.075626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:46.075718 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:46.075842 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:46.077769 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.078387 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.079763 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.080108 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.080254 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:46.083725 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:46.083729 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:46.083731 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:46.083761 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:46.085801 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:46.086362 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:46.086369 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:46.086381 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:46.086497 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:46.087029 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:46.087033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:46.087035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:46.087356 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:46.087498 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:46.088235 (+ 737us) server_negotiation.cc:299] Negotiation successful
0810 01:46:46.088420 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 01:46:46.089620 12310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:46.073476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46807 (local address 127.9.77.129:42859)
0810 01:46:46.074056 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:46.074117 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:46.075286 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:46.076187 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:46.076195 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:46.076746 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:46:46.077581 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:46.077600 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.078537 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.078541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:46.079644 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:46.079658 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.080355 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.080360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:46.080670 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:46.081593 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:46:46.081612 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:46:46.083504 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:46:46.085937 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:46.085943 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:46.085958 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:46.086247 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:46.086655 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:46.086659 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:46.086661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:46.086885 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:46.087557 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:46:46.087564 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:46:46.087928 (+ 364us) client_negotiation.cc:769] Sending connection context
0810 01:46:46.088185 (+ 257us) client_negotiation.cc:240] Negotiation successful
0810 01:46:46.088589 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":165,"threads_started":1}
I20260810 01:46:46.091156 12468 heartbeater.cc:347] Connected to a master server at 127.9.77.190:46807
I20260810 01:46:46.091336 12468 heartbeater.cc:464] Registering TS with master...
I20260810 01:46:46.092060 12468 heartbeater.cc:511] Master 127.9.77.190:46807 requested a full tablet report, sending...
I20260810 01:46:46.094197 12163 ts_manager.cc:194] Registered new tserver with Master: 40a6de2a6f85463998b46a214b210bb2 (127.9.77.129:39511)
I20260810 01:46:46.095592 12163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.129@KRBTEST.COM'} at 127.9.77.129:42859
I20260810 01:46:46.097142 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 12275
I20260810 01:46:46.097286 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance
I20260810 01:46:46.097695 9526 external_mini_cluster.cc:1428] Setting key df80164fa157f0358c96a1c97e12c77b
I20260810 01:46:46.103003 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:46.094153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:33449 (local address 127.9.77.190:46807)
0810 01:46:46.094293 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:46.094297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:46.094472 (+ 175us) server_negotiation.cc:407] Connection header received
0810 01:46:46.094623 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:46.094626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:46.094680 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:46.094775 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:46.095281 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.095957 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.097185 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.097519 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.097638 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:46.099146 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:46.099149 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:46.099151 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:46.099175 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:46.101012 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:46.101472 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:46.101478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:46.101482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:46.101590 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:46.101882 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:46.101887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:46.101892 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:46.102165 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:46.102277 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:46.102594 (+ 317us) server_negotiation.cc:299] Negotiation successful
0810 01:46:46.102765 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 01:46:46.103008 12479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:46.094037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46807 (local address 127.9.77.129:33449)
0810 01:46:46.094385 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:46.094398 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:46.094518 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:46.094790 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:46.094794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:46.095047 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:46:46.095164 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:46.095170 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.096114 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.096118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:46.097052 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:46.097063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.097657 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.097661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:46.097754 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:46.098467 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:46:46.098480 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:46:46.099007 (+ 527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:46:46.101142 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:46.101145 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:46.101147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:46.101363 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:46.101694 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:46.101696 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:46.101698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:46.101787 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:46.102289 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:46:46.102292 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:46:46.102400 (+ 108us) client_negotiation.cc:769] Sending connection context
0810 01:46:46.102507 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 01:46:46.102679 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.9.77.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
I20260810 01:46:46.137285 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:0
--local_ip_for_outbound_sockets=127.9.77.130
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:46807
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:42529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:46:46.276512 12483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:46:46.276845 12483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:46:46.276935 12483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:46:46.281320 12483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:46:46.281404 12483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:46:46.281450 12483 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 01:46:46.281466 12483 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 01:46:46.281531 12483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:46:46.287649 12483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:42529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.77.190:46807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:46:46.289269 12483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:46:46.290690 12483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:46:46.302845 12491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:46.302929 12492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:46.303062 12495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:46.303107 12483 server_base.cc:1034] running on GCE node
I20260810 01:46:46.303714 12483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:46:46.304572 12483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:46:46.305783 12483 hybrid_clock.cc:648] HybridClock initialized: now 1786326406305764 us; error 30 us; skew 500 ppm
Aug 10 01:46:46 dist-test-slave-kvfs krb5kdc[12097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326406, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:46:46.309563 12483 init.cc:377] Logged in from keytab as kudu/127.9.77.130@KRBTEST.COM (short username kudu)
I20260810 01:46:46.310732 12483 webserver.cc:460] Webserver started at http://127.9.77.130:36707/ using document root <none> and password file <none>
I20260810 01:46:46.311470 12483 fs_manager.cc:359] Metadata directory not provided
I20260810 01:46:46.311540 12483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:46:46.311808 12483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:46:46.314555 12483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-1/data/instance:
uuid: "9e29676918e746de84c4e8b62f658f35"
format_stamp: "Formatted at 2026-08-10 01:46:46 on dist-test-slave-kvfs"
server_key: "5bb5c96d638a56250abb3d91166a82cf"
server_key_iv: "9e1704a265210a165ec81d9f4a10f805"
server_key_version: "encryptionkey@0"
I20260810 01:46:46.315256 12483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance:
uuid: "9e29676918e746de84c4e8b62f658f35"
format_stamp: "Formatted at 2026-08-10 01:46:46 on dist-test-slave-kvfs"
server_key: "5bb5c96d638a56250abb3d91166a82cf"
server_key_iv: "9e1704a265210a165ec81d9f4a10f805"
server_key_version: "encryptionkey@0"
I20260810 01:46:46.320165 12483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 01:46:46.323349 12510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:46.325121 12483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:46:46.325294 12483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "9e29676918e746de84c4e8b62f658f35"
format_stamp: "Formatted at 2026-08-10 01:46:46 on dist-test-slave-kvfs"
server_key: "5bb5c96d638a56250abb3d91166a82cf"
server_key_iv: "9e1704a265210a165ec81d9f4a10f805"
server_key_version: "encryptionkey@0"
I20260810 01:46:46.325449 12483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:46:46.339701 12483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:46:46.343806 12483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:46:46.344152 12483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:46:46.346107 12483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:46:46.346199 12483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:46.346288 12483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:46:46.346342 12483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 01:46:46 dist-test-slave-kvfs krb5kdc[12097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326406, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:46.360000 12483 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:45423
I20260810 01:46:46.360066 12683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:45423 every 8 connection(s)
I20260810 01:46:46.361486 12483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:46:46.363838 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:46.347227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:35005 (local address 127.9.77.190:46807)
0810 01:46:46.347439 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:46.347459 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:46.348768 (+ 1309us) server_negotiation.cc:407] Connection header received
0810 01:46:46.350089 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:46.350093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:46.350172 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:46.350306 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:46.352461 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.353498 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.354935 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.355375 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.355518 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:46.359176 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:46.359179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:46.359182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:46.359218 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:46.361149 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:46.361883 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:46.361886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:46.361888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:46.362021 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:46.362469 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:46.362473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:46.362475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:46.362758 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:46.362880 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:46.363537 (+ 657us) server_negotiation.cc:299] Negotiation successful
0810 01:46:46.363677 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 01:46:46.364912 12525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:46.347795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46807 (local address 127.9.77.130:35005)
0810 01:46:46.348557 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:46.348618 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:46.349774 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:46.350613 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:46.350626 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:46.351204 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:46:46.352189 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:46.352222 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.353679 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.353683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:46.354809 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:46.354819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.355606 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.355611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:46.355799 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:46.356697 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:46:46.356716 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:46:46.358910 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:46:46.361296 (+ 2386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:46.361302 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:46.361319 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:46.361736 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:46.362153 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:46.362156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:46.362158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:46.362347 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:46.362896 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:46:46.362904 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:46:46.363278 (+ 374us) client_negotiation.cc:769] Sending connection context
0810 01:46:46.363471 (+ 193us) client_negotiation.cc:240] Negotiation successful
0810 01:46:46.363828 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":227,"threads_started":1}
I20260810 01:46:46.366534 12684 heartbeater.cc:347] Connected to a master server at 127.9.77.190:46807
I20260810 01:46:46.366767 12684 heartbeater.cc:464] Registering TS with master...
I20260810 01:46:46.367462 12684 heartbeater.cc:511] Master 127.9.77.190:46807 requested a full tablet report, sending...
I20260810 01:46:46.368829 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 12483
I20260810 01:46:46.369010 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance
I20260810 01:46:46.369011 12156 ts_manager.cc:194] Registered new tserver with Master: 9e29676918e746de84c4e8b62f658f35 (127.9.77.130:45423)
I20260810 01:46:46.369328 9526 external_mini_cluster.cc:1428] Setting key 719fe34749a07c0f209117bb3c40a8e5
I20260810 01:46:46.369962 12156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.130@KRBTEST.COM'} at 127.9.77.130:35005
I20260810 01:46:46.380205 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:46.369548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:40001 (local address 127.9.77.190:46807)
0810 01:46:46.369678 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:46.369682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:46.370358 (+ 676us) server_negotiation.cc:407] Connection header received
0810 01:46:46.370570 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:46.370574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:46.370626 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:46.370727 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:46.372719 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.373553 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.374588 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.374842 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.374966 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:46.376443 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:46.376447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:46.376450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:46.376487 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:46.378214 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:46.378706 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:46.378710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:46.378712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:46.378832 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:46.379137 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:46.379141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:46.379143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:46.379427 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:46.379544 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:46.379836 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 01:46:46.379970 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 01:46:46.380215 12695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:46.369541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46807 (local address 127.9.77.130:40001)
0810 01:46:46.370268 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:46.370287 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:46.370437 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:46.372139 (+ 1702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:46.372142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:46.372431 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:46:46.372570 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:46.372578 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.373760 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.373764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:46.374424 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:46.374434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.374976 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.374980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:46.375082 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:46.375765 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:46:46.375779 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:46:46.376291 (+ 512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:46:46.378358 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:46.378361 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:46.378364 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:46.378596 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:46.378936 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:46.378939 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:46.378942 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:46.379038 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:46.379530 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:46:46.379533 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:46:46.379646 (+ 113us) client_negotiation.cc:769] Sending connection context
0810 01:46:46.379762 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 01:46:46.379952 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":466,"threads_started":1}
WARNING: no policy specified for kudu/127.9.77.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
I20260810 01:46:46.408775 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:0
--local_ip_for_outbound_sockets=127.9.77.131
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:46807
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:42529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:46:46.549618 12700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:46:46.550002 12700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:46:46.550057 12700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:46:46.554386 12700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:46:46.554468 12700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:46:46.554513 12700 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 01:46:46.554554 12700 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 01:46:46.554618 12700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:46:46.560961 12700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:42529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.77.190:46807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:46:46.562484 12700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:46:46.563879 12700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:46:46.576397 12709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:46.576419 12707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:46.576397 12711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:46.576879 12700 server_base.cc:1034] running on GCE node
I20260810 01:46:46.577402 12700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:46:46.578256 12700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:46:46.579468 12700 hybrid_clock.cc:648] HybridClock initialized: now 1786326406579452 us; error 29 us; skew 500 ppm
Aug 10 01:46:46 dist-test-slave-kvfs krb5kdc[12097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326406, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:46:46.583071 12700 init.cc:377] Logged in from keytab as kudu/127.9.77.131@KRBTEST.COM (short username kudu)
I20260810 01:46:46.584331 12700 webserver.cc:460] Webserver started at http://127.9.77.131:44277/ using document root <none> and password file <none>
I20260810 01:46:46.585069 12700 fs_manager.cc:359] Metadata directory not provided
I20260810 01:46:46.585170 12700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:46:46.585460 12700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:46:46.588131 12700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-2/data/instance:
uuid: "e7a50c0229de40a6bc28bd32ea6649ef"
format_stamp: "Formatted at 2026-08-10 01:46:46 on dist-test-slave-kvfs"
server_key: "00ff183fc2f53cdbccaffdae783544d0"
server_key_iv: "24b587ec5aae70213ec9ad3a5c9c530e"
server_key_version: "encryptionkey@0"
I20260810 01:46:46.588883 12700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance:
uuid: "e7a50c0229de40a6bc28bd32ea6649ef"
format_stamp: "Formatted at 2026-08-10 01:46:46 on dist-test-slave-kvfs"
server_key: "00ff183fc2f53cdbccaffdae783544d0"
server_key_iv: "24b587ec5aae70213ec9ad3a5c9c530e"
server_key_version: "encryptionkey@0"
I20260810 01:46:46.593966 12700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 01:46:46.598264 12720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:46.600423 12700 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 01:46:46.600579 12700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "e7a50c0229de40a6bc28bd32ea6649ef"
format_stamp: "Formatted at 2026-08-10 01:46:46 on dist-test-slave-kvfs"
server_key: "00ff183fc2f53cdbccaffdae783544d0"
server_key_iv: "24b587ec5aae70213ec9ad3a5c9c530e"
server_key_version: "encryptionkey@0"
I20260810 01:46:46.600741 12700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:46:46.618746 12700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:46:46.622776 12700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:46:46.622979 12700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:46:46.624912 12700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:46:46.624969 12700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:46.625041 12700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:46:46.625075 12700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 01:46:46 dist-test-slave-kvfs krb5kdc[12097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326406, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:46.637854 12700 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:35103
I20260810 01:46:46.637934 12887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:35103 every 8 connection(s)
I20260810 01:46:46.639246 12700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:46:46.642356 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:46.625808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:37723 (local address 127.9.77.190:46807)
0810 01:46:46.626010 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:46.626016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:46.627038 (+ 1022us) server_negotiation.cc:407] Connection header received
0810 01:46:46.628326 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:46.628330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:46.628412 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:46.628520 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:46.630880 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.631886 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.633457 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.633874 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.634029 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:46.637825 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:46.637832 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:46.637836 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:46.637889 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:46.639737 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:46.640541 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:46.640545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:46.640547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:46.640668 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:46.641120 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:46.641123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:46.641125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:46.641387 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:46.641491 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:46.642068 (+ 577us) server_negotiation.cc:299] Negotiation successful
0810 01:46:46.642203 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 01:46:46.643397 12731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:46.626301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46807 (local address 127.9.77.131:37723)
0810 01:46:46.626840 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:46.626899 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:46.627950 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:46.629140 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:46.629153 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:46.629729 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:46:46.630643 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:46.630672 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.632076 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.632082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:46.633303 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:46.633314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.634090 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.634095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:46.634338 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:46.635126 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:46:46.635157 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:46:46.637541 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:46:46.639913 (+ 2372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:46.639919 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:46.639937 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:46.640374 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:46.640775 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:46.640778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:46.640779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:46.640958 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:46.641498 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:46:46.641504 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:46:46.641797 (+ 293us) client_negotiation.cc:769] Sending connection context
0810 01:46:46.641992 (+ 195us) client_negotiation.cc:240] Negotiation successful
0810 01:46:46.642313 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":156,"threads_started":1}
I20260810 01:46:46.644969 12888 heartbeater.cc:347] Connected to a master server at 127.9.77.190:46807
I20260810 01:46:46.645169 12888 heartbeater.cc:464] Registering TS with master...
I20260810 01:46:46.645767 12888 heartbeater.cc:511] Master 127.9.77.190:46807 requested a full tablet report, sending...
I20260810 01:46:46.647111 12156 ts_manager.cc:194] Registered new tserver with Master: e7a50c0229de40a6bc28bd32ea6649ef (127.9.77.131:35103)
I20260810 01:46:46.648015 12156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.131@KRBTEST.COM'} at 127.9.77.131:37723
I20260810 01:46:46.649554 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 12700
I20260810 01:46:46.649698 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance
I20260810 01:46:46.650100 9526 external_mini_cluster.cc:1428] Setting key 2ad53215e8df16f1e685d784521f6efa
I20260810 01:46:46.652693 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 01:46:46.656327 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:46.647876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:44995 (local address 127.9.77.190:46807)
0810 01:46:46.648043 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:46.648051 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:46.648279 (+ 228us) server_negotiation.cc:407] Connection header received
0810 01:46:46.648540 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:46.648544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:46.648596 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:46.648694 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:46.649305 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.650094 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.651213 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.651451 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.651584 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:46.652940 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:46.652944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:46.652947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:46.652973 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:46.654495 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:46.654973 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:46.654977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:46.654979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:46.655070 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:46.655376 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:46.655379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:46.655381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:46.655588 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:46.655705 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:46.655994 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 01:46:46.656156 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 01:46:46.656337 12899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:46.647809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46807 (local address 127.9.77.131:44995)
0810 01:46:46.648175 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:46.648194 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:46.648431 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:46.648723 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:46.648725 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:46.649011 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:46:46.649174 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:46.649184 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.650268 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.650273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:46.651113 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:46.651124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.651578 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.651581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:46.651683 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:46.652377 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:46:46.652398 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:46:46.652799 (+ 401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:46:46.654622 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:46.654625 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:46.654628 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:46.654850 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:46.655165 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:46.655169 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:46.655171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:46.655265 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:46.655693 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:46:46.655696 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:46:46.655806 (+ 110us) client_negotiation.cc:769] Sending connection context
0810 01:46:46.655922 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 01:46:46.656128 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":107,"threads_started":1}
Aug 10 01:46:46 dist-test-slave-kvfs krb5kdc[12097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 01:46:46 dist-test-slave-kvfs krb5kdc[12097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:46.687937 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:46.678295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41254 (local address 127.9.77.190:46807)
0810 01:46:46.678496 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:46.678501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:46.678517 (+ 16us) server_negotiation.cc:407] Connection header received
0810 01:46:46.678576 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:46.678579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:46.678628 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:46.678762 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:46.679605 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.680271 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.681207 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.681457 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.681578 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:46.684464 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:46.684468 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:46.684470 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:46.684503 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:46.686034 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:46.686510 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:46.686514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:46.686516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:46.686609 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:46.686904 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:46.686907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:46.686909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:46.687138 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:46.687298 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:46.687562 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 01:46:46.687720 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 01:46:46.691782 12156 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 01:46:46.695676 12156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:46:46.713435 12923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:46.707516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54418 (local address 127.9.77.131:35103)
0810 01:46:46.708080 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:46.708094 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:46.708125 (+ 31us) server_negotiation.cc:407] Connection header received
0810 01:46:46.708227 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:46.708232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:46.708493 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:46.708674 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:46:46.709091 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.710033 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.711592 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.712319 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.712451 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:46.712682 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:46.713115 (+ 433us) server_negotiation.cc:299] Negotiation successful
0810 01:46:46.713226 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":154,"threads_started":1}
I20260810 01:46:46.713450 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:46.706653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:35103 (local address 127.0.0.1:54418)
0810 01:46:46.707677 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:46.707692 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:46.707804 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:46.708724 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:46.708733 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:46.708761 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:46:46.708976 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:46.708986 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.710183 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.710186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:46.711461 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:46.711471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.712751 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.712755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:46.712887 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:46.712935 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 01:46:46.713067 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 01:46:46.713211 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":959,"mutex_wait_us":3,"thread_start_us":141,"threads_started":1}
I20260810 01:46:46.714511 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:46.707280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:45423 (local address 127.0.0.1:33368)
0810 01:46:46.708101 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:46.708115 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:46.708212 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:46.708882 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:46.708885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:46.708975 (+ 90us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:46:46.709118 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:46.709128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.710662 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.710665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:46.712276 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:46.712288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.714002 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.714005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:46.714114 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:46.714144 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 01:46:46.714244 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 01:46:46.714336 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":190,"spinlock_wait_cycles":1920,"thread_start_us":125,"threads_started":1}
I20260810 01:46:46.715073 12924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:46.707819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33368 (local address 127.9.77.130:45423)
0810 01:46:46.708429 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:46.708439 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:46.708460 (+ 21us) server_negotiation.cc:407] Connection header received
0810 01:46:46.708532 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:46.708536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:46.708754 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:46.708918 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:46:46.709305 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.710516 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.712420 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.713281 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.713395 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:46.713579 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:46.714796 (+ 1217us) server_negotiation.cc:299] Negotiation successful
0810 01:46:46.714910 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":139,"threads_started":1}
I20260810 01:46:46.715334 12913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:46.706494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:39511 (local address 127.0.0.1:35166)
0810 01:46:46.707048 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:46.707107 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:46.707268 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:46.708881 (+ 1613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:46.708884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:46.708956 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:46:46.709138 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:46.709146 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.710662 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.710665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:46.712276 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:46.712288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.714922 (+ 2634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.714928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:46.715019 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:46.715045 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 01:46:46.715128 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 01:46:46.715212 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":242,"thread_start_us":115,"threads_started":1}
I20260810 01:46:46.715880 12919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:46.706753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35166 (local address 127.9.77.129:39511)
0810 01:46:46.708115 (+ 1362us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:46.708122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:46.708146 (+ 24us) server_negotiation.cc:407] Connection header received
0810 01:46:46.708226 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:46.708232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:46.708493 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:46.708674 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:46:46.709269 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.710516 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.713658 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.714468 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.714583 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:46.714719 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:46.715610 (+ 891us) server_negotiation.cc:299] Negotiation successful
0810 01:46:46.715723 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1223,"thread_start_us":90,"threads_started":1}
I20260810 01:46:46.718043 12583 tablet_service.cc:1467] Processing CreateTablet for tablet 4818d04f41f043648a20694827dd362f (DEFAULT_TABLE table=test-table [id=03aaa2032725434498d755e565a98b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:46:46.718043 12786 tablet_service.cc:1467] Processing CreateTablet for tablet 4818d04f41f043648a20694827dd362f (DEFAULT_TABLE table=test-table [id=03aaa2032725434498d755e565a98b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:46:46.718312 12375 tablet_service.cc:1467] Processing CreateTablet for tablet 4818d04f41f043648a20694827dd362f (DEFAULT_TABLE table=test-table [id=03aaa2032725434498d755e565a98b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:46:46.719362 12375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4818d04f41f043648a20694827dd362f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:46.719368 12583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4818d04f41f043648a20694827dd362f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:46.719383 12786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4818d04f41f043648a20694827dd362f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:46.726944 12926 tablet_bootstrap.cc:492] T 4818d04f41f043648a20694827dd362f P 40a6de2a6f85463998b46a214b210bb2: Bootstrap starting.
I20260810 01:46:46.728904 12928 tablet_bootstrap.cc:492] T 4818d04f41f043648a20694827dd362f P 9e29676918e746de84c4e8b62f658f35: Bootstrap starting.
I20260810 01:46:46.728868 12927 tablet_bootstrap.cc:492] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef: Bootstrap starting.
I20260810 01:46:46.730026 12926 tablet_bootstrap.cc:654] T 4818d04f41f043648a20694827dd362f P 40a6de2a6f85463998b46a214b210bb2: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:46.731462 12928 tablet_bootstrap.cc:654] T 4818d04f41f043648a20694827dd362f P 9e29676918e746de84c4e8b62f658f35: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:46.731462 12927 tablet_bootstrap.cc:654] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:46.731806 12926 log.cc:824] T 4818d04f41f043648a20694827dd362f P 40a6de2a6f85463998b46a214b210bb2: Log is configured to *not* fsync() on all Append() calls
I20260810 01:46:46.732437 12927 log.cc:824] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef: Log is configured to *not* fsync() on all Append() calls
I20260810 01:46:46.732437 12928 log.cc:824] T 4818d04f41f043648a20694827dd362f P 9e29676918e746de84c4e8b62f658f35: Log is configured to *not* fsync() on all Append() calls
I20260810 01:46:46.734362 12928 tablet_bootstrap.cc:492] T 4818d04f41f043648a20694827dd362f P 9e29676918e746de84c4e8b62f658f35: No bootstrap required, opened a new log
I20260810 01:46:46.734408 12926 tablet_bootstrap.cc:492] T 4818d04f41f043648a20694827dd362f P 40a6de2a6f85463998b46a214b210bb2: No bootstrap required, opened a new log
I20260810 01:46:46.734408 12927 tablet_bootstrap.cc:492] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef: No bootstrap required, opened a new log
I20260810 01:46:46.734627 12928 ts_tablet_manager.cc:1397] T 4818d04f41f043648a20694827dd362f P 9e29676918e746de84c4e8b62f658f35: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 01:46:46.734627 12926 ts_tablet_manager.cc:1397] T 4818d04f41f043648a20694827dd362f P 40a6de2a6f85463998b46a214b210bb2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260810 01:46:46.734627 12927 ts_tablet_manager.cc:1397] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 01:46:46.739198 12928 raft_consensus.cc:348] T 4818d04f41f043648a20694827dd362f P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } }
I20260810 01:46:46.739598 12928 raft_consensus.cc:374] T 4818d04f41f043648a20694827dd362f P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:46.739665 12928 raft_consensus.cc:729] T 4818d04f41f043648a20694827dd362f P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e29676918e746de84c4e8b62f658f35, State: Initialized, Role: FOLLOWER
I20260810 01:46:46.739519 12927 raft_consensus.cc:348] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } }
I20260810 01:46:46.739832 12927 raft_consensus.cc:374] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:46.739912 12927 raft_consensus.cc:729] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a50c0229de40a6bc28bd32ea6649ef, State: Initialized, Role: FOLLOWER
I20260810 01:46:46.740409 12927 consensus_queue.cc:260] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } }
I20260810 01:46:46.740410 12928 consensus_queue.cc:260] T 4818d04f41f043648a20694827dd362f P 9e29676918e746de84c4e8b62f658f35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } }
I20260810 01:46:46.741082 12926 raft_consensus.cc:348] T 4818d04f41f043648a20694827dd362f P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } }
I20260810 01:46:46.741375 12926 raft_consensus.cc:374] T 4818d04f41f043648a20694827dd362f P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:46.741436 12926 raft_consensus.cc:729] T 4818d04f41f043648a20694827dd362f P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40a6de2a6f85463998b46a214b210bb2, State: Initialized, Role: FOLLOWER
I20260810 01:46:46.741737 12927 ts_tablet_manager.cc:1428] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 01:46:46.741737 12928 ts_tablet_manager.cc:1428] T 4818d04f41f043648a20694827dd362f P 9e29676918e746de84c4e8b62f658f35: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 01:46:46.742195 12888 heartbeater.cc:503] Master 127.9.77.190:46807 was elected leader, sending a full tablet report...
I20260810 01:46:46.742197 12684 heartbeater.cc:503] Master 127.9.77.190:46807 was elected leader, sending a full tablet report...
I20260810 01:46:46.741922 12926 consensus_queue.cc:260] T 4818d04f41f043648a20694827dd362f P 40a6de2a6f85463998b46a214b210bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } }
I20260810 01:46:46.744704 12926 ts_tablet_manager.cc:1428] T 4818d04f41f043648a20694827dd362f P 40a6de2a6f85463998b46a214b210bb2: Time spent starting tablet: real 0.010s user 0.004s sys 0.004s
I20260810 01:46:46.745127 12468 heartbeater.cc:503] Master 127.9.77.190:46807 was elected leader, sending a full tablet report...
W20260810 01:46:46.838622 12470 tablet.cc:2369] T 4818d04f41f043648a20694827dd362f P 40a6de2a6f85463998b46a214b210bb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 01:46:46.853446 12933 raft_consensus.cc:482] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:46:46.853888 12933 raft_consensus.cc:504] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } }
I20260810 01:46:46.855523 12933 leader_election.cc:288] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40a6de2a6f85463998b46a214b210bb2 (127.9.77.129:39511), 9e29676918e746de84c4e8b62f658f35 (127.9.77.130:45423)
I20260810 01:46:46.860684 12919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:46.856100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:43171 (local address 127.9.77.129:39511)
0810 01:46:46.856249 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:46.856255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:46.856278 (+ 23us) server_negotiation.cc:407] Connection header received
0810 01:46:46.856449 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:46.856453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:46.856520 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:46.856637 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:46:46.857034 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.857936 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.859349 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.860074 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.860175 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:46.860231 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:46.860471 (+ 240us) server_negotiation.cc:299] Negotiation successful
0810 01:46:46.860552 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 01:46:46.860687 12731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:46.855951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:39511 (local address 127.9.77.131:43171)
0810 01:46:46.856118 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:46.856141 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:46.856274 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:46.856690 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:46.856694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:46.856718 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:46:46.856877 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:46.856885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.858066 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.858069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:46.859200 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:46.859210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.860185 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.860189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:46.860302 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:46.860350 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 01:46:46.860435 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 01:46:46.860522 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 01:46:46.861331 12937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:46.855951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:45423 (local address 127.9.77.131:38161)
0810 01:46:46.856561 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:46.856577 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:46.856700 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:46.857030 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:46.857033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:46.857053 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:46:46.857681 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:46.857690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.858825 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.858828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:46.859873 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:46.859883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.860828 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.860833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:46.860947 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:46.860981 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 01:46:46.861104 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 01:46:46.861174 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":168,"thread_start_us":145,"threads_started":1}
I20260810 01:46:46.861356 12924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:46.856100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:38161 (local address 127.9.77.130:45423)
0810 01:46:46.856270 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:46.856276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:46.856705 (+ 429us) server_negotiation.cc:407] Connection header received
0810 01:46:46.856785 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:46.856788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:46.856877 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:46.856972 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:46:46.857865 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.858672 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.860063 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.860675 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.860822 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:46.860880 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:46.861144 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 01:46:46.861225 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 01:46:46.861613 12400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4818d04f41f043648a20694827dd362f" candidate_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40a6de2a6f85463998b46a214b210bb2" is_pre_election: true
I20260810 01:46:46.862083 12400 raft_consensus.cc:2454] T 4818d04f41f043648a20694827dd362f P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7a50c0229de40a6bc28bd32ea6649ef in term 0.
I20260810 01:46:46.862195 12611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4818d04f41f043648a20694827dd362f" candidate_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e29676918e746de84c4e8b62f658f35" is_pre_election: true
I20260810 01:46:46.862577 12611 raft_consensus.cc:2454] T 4818d04f41f043648a20694827dd362f P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7a50c0229de40a6bc28bd32ea6649ef in term 0.
I20260810 01:46:46.862727 12723 leader_election.cc:302] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40a6de2a6f85463998b46a214b210bb2, e7a50c0229de40a6bc28bd32ea6649ef; no voters:
W20260810 01:46:46.862982 12685 tablet.cc:2369] T 4818d04f41f043648a20694827dd362f P 9e29676918e746de84c4e8b62f658f35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 01:46:46.863130 12933 raft_consensus.cc:2781] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:46:46.863227 12933 raft_consensus.cc:482] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:46:46.863288 12933 raft_consensus.cc:3037] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:46.864786 12933 raft_consensus.cc:504] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } }
I20260810 01:46:46.865187 12933 leader_election.cc:288] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef [CANDIDATE]: Term 1 election: Requested vote from peers 40a6de2a6f85463998b46a214b210bb2 (127.9.77.129:39511), 9e29676918e746de84c4e8b62f658f35 (127.9.77.130:45423)
I20260810 01:46:46.865634 12611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4818d04f41f043648a20694827dd362f" candidate_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e29676918e746de84c4e8b62f658f35"
I20260810 01:46:46.865648 12400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4818d04f41f043648a20694827dd362f" candidate_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40a6de2a6f85463998b46a214b210bb2"
I20260810 01:46:46.865759 12611 raft_consensus.cc:3037] T 4818d04f41f043648a20694827dd362f P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:46.865792 12400 raft_consensus.cc:3037] T 4818d04f41f043648a20694827dd362f P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:46.867172 12611 raft_consensus.cc:2454] T 4818d04f41f043648a20694827dd362f P 9e29676918e746de84c4e8b62f658f35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7a50c0229de40a6bc28bd32ea6649ef in term 1.
I20260810 01:46:46.867194 12400 raft_consensus.cc:2454] T 4818d04f41f043648a20694827dd362f P 40a6de2a6f85463998b46a214b210bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7a50c0229de40a6bc28bd32ea6649ef in term 1.
I20260810 01:46:46.867565 12726 leader_election.cc:302] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e29676918e746de84c4e8b62f658f35, e7a50c0229de40a6bc28bd32ea6649ef; no voters:
I20260810 01:46:46.867766 12933 raft_consensus.cc:2781] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:46:46.868006 12933 raft_consensus.cc:686] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef [term 1 LEADER]: Becoming Leader. State: Replica: e7a50c0229de40a6bc28bd32ea6649ef, State: Running, Role: LEADER
I20260810 01:46:46.868571 12933 consensus_queue.cc:237] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } }
I20260810 01:46:46.872550 12163 catalog_manager.cc:5495] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef reported cstate change: term changed from 0 to 1, leader changed from <none> to e7a50c0229de40a6bc28bd32ea6649ef (127.9.77.131). New cstate: current_term: 1 leader_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } health_report { overall_health: UNKNOWN } } }
W20260810 01:46:46.890619 12891 tablet.cc:2369] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 01:46:46 dist-test-slave-kvfs krb5kdc[12097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326405, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:46.952494 12950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:46.942599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46807 (local address 127.0.0.1:41262)
0810 01:46:46.942976 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:46.942992 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:46.943183 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:46.943478 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:46.943481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:46.944080 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:46:46.944215 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:46.944224 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.945143 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.945148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:46.945815 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:46.945825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.946307 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.946311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:46.946411 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:46.947290 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:46:46.947304 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:46:46.949165 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:46:46.950779 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:46.950789 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:46.950795 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:46.951057 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:46.951337 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:46.951340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:46.951342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:46.951435 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:46.951802 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:46:46.951811 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:46:46.951938 (+ 127us) client_negotiation.cc:769] Sending connection context
0810 01:46:46.952117 (+ 179us) client_negotiation.cc:240] Negotiation successful
0810 01:46:46.952267 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":140,"threads_started":1}
I20260810 01:46:46.952493 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:46.942730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41262 (local address 127.9.77.190:46807)
0810 01:46:46.942921 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:46.942926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:46.943096 (+ 170us) server_negotiation.cc:407] Connection header received
0810 01:46:46.943314 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:46.943317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:46.943360 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:46.943455 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:46.944350 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.945024 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.945952 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:46.946187 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:46.946299 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:46.949316 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:46.949319 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:46.949321 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:46.949345 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:46.950675 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:46.951147 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:46.951149 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:46.951151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:46.951227 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:46.951517 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:46.951519 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:46.951520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:46.951710 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:46.951784 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:46.952167 (+ 383us) server_negotiation.cc:299] Negotiation successful
0810 01:46:46.952353 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":61}
I20260810 01:46:46.957469 12163 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:41262:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 01:46:46.958439 12163 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:46:46.964320 12583 tablet_service.cc:1467] Processing CreateTablet for tablet 1f27eccf1a494667bcbe2a6c28abac4c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2abf312d4457417ebadba0d6cd94c30d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 01:46:46.964284 12786 tablet_service.cc:1467] Processing CreateTablet for tablet 1f27eccf1a494667bcbe2a6c28abac4c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2abf312d4457417ebadba0d6cd94c30d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 01:46:46.964278 12375 tablet_service.cc:1467] Processing CreateTablet for tablet 1f27eccf1a494667bcbe2a6c28abac4c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2abf312d4457417ebadba0d6cd94c30d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 01:46:46.965214 12583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f27eccf1a494667bcbe2a6c28abac4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:46.965215 12786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f27eccf1a494667bcbe2a6c28abac4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:46.965214 12375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f27eccf1a494667bcbe2a6c28abac4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:46.968848 12928 tablet_bootstrap.cc:492] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35: Bootstrap starting.
I20260810 01:46:46.968848 12926 tablet_bootstrap.cc:492] T 1f27eccf1a494667bcbe2a6c28abac4c P 40a6de2a6f85463998b46a214b210bb2: Bootstrap starting.
I20260810 01:46:46.970008 12926 tablet_bootstrap.cc:654] T 1f27eccf1a494667bcbe2a6c28abac4c P 40a6de2a6f85463998b46a214b210bb2: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:46.970007 12928 tablet_bootstrap.cc:654] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:46.970772 12927 tablet_bootstrap.cc:492] T 1f27eccf1a494667bcbe2a6c28abac4c P e7a50c0229de40a6bc28bd32ea6649ef: Bootstrap starting.
I20260810 01:46:46.971606 12926 tablet_bootstrap.cc:492] T 1f27eccf1a494667bcbe2a6c28abac4c P 40a6de2a6f85463998b46a214b210bb2: No bootstrap required, opened a new log
I20260810 01:46:46.971727 12926 ts_tablet_manager.cc:1397] T 1f27eccf1a494667bcbe2a6c28abac4c P 40a6de2a6f85463998b46a214b210bb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 01:46:46.971918 12927 tablet_bootstrap.cc:654] T 1f27eccf1a494667bcbe2a6c28abac4c P e7a50c0229de40a6bc28bd32ea6649ef: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:46.971925 12928 tablet_bootstrap.cc:492] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35: No bootstrap required, opened a new log
I20260810 01:46:46.972014 12928 ts_tablet_manager.cc:1397] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 01:46:46.972349 12926 raft_consensus.cc:348] T 1f27eccf1a494667bcbe2a6c28abac4c P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:46.972507 12926 raft_consensus.cc:374] T 1f27eccf1a494667bcbe2a6c28abac4c P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:46.972551 12926 raft_consensus.cc:729] T 1f27eccf1a494667bcbe2a6c28abac4c P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40a6de2a6f85463998b46a214b210bb2, State: Initialized, Role: FOLLOWER
I20260810 01:46:46.972604 12928 raft_consensus.cc:348] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:46.972740 12928 raft_consensus.cc:374] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:46.972785 12928 raft_consensus.cc:729] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e29676918e746de84c4e8b62f658f35, State: Initialized, Role: FOLLOWER
I20260810 01:46:46.972766 12926 consensus_queue.cc:260] T 1f27eccf1a494667bcbe2a6c28abac4c P 40a6de2a6f85463998b46a214b210bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:46.972927 12928 consensus_queue.cc:260] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:46.973110 12926 ts_tablet_manager.cc:1428] T 1f27eccf1a494667bcbe2a6c28abac4c P 40a6de2a6f85463998b46a214b210bb2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 01:46:46.973134 12936 tablet_replica.cc:439] T 1f27eccf1a494667bcbe2a6c28abac4c P 40a6de2a6f85463998b46a214b210bb2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } } }
I20260810 01:46:46.973330 12936 tablet_replica.cc:442] T 1f27eccf1a494667bcbe2a6c28abac4c P 40a6de2a6f85463998b46a214b210bb2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 01:46:46.973248 12935 tablet_replica.cc:439] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } } }
I20260810 01:46:46.973407 12935 tablet_replica.cc:442] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 01:46:46.973650 12928 ts_tablet_manager.cc:1428] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 01:46:46.974443 12927 tablet_bootstrap.cc:492] T 1f27eccf1a494667bcbe2a6c28abac4c P e7a50c0229de40a6bc28bd32ea6649ef: No bootstrap required, opened a new log
I20260810 01:46:46.974539 12927 ts_tablet_manager.cc:1397] T 1f27eccf1a494667bcbe2a6c28abac4c P e7a50c0229de40a6bc28bd32ea6649ef: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260810 01:46:46.975090 12927 raft_consensus.cc:348] T 1f27eccf1a494667bcbe2a6c28abac4c P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:46.975244 12927 raft_consensus.cc:374] T 1f27eccf1a494667bcbe2a6c28abac4c P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:46.975286 12927 raft_consensus.cc:729] T 1f27eccf1a494667bcbe2a6c28abac4c P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a50c0229de40a6bc28bd32ea6649ef, State: Initialized, Role: FOLLOWER
I20260810 01:46:46.975441 12927 consensus_queue.cc:260] T 1f27eccf1a494667bcbe2a6c28abac4c P e7a50c0229de40a6bc28bd32ea6649ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:46.975724 12927 ts_tablet_manager.cc:1428] T 1f27eccf1a494667bcbe2a6c28abac4c P e7a50c0229de40a6bc28bd32ea6649ef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 01:46:46.976104 12933 tablet_replica.cc:439] T 1f27eccf1a494667bcbe2a6c28abac4c P e7a50c0229de40a6bc28bd32ea6649ef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } } }
I20260810 01:46:46.976276 12933 tablet_replica.cc:442] T 1f27eccf1a494667bcbe2a6c28abac4c P e7a50c0229de40a6bc28bd32ea6649ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 01:46:47.038995 12935 raft_consensus.cc:482] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:46:47.039207 12935 raft_consensus.cc:504] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:47.040735 12935 leader_election.cc:288] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40a6de2a6f85463998b46a214b210bb2 (127.9.77.129:39511), e7a50c0229de40a6bc28bd32ea6649ef (127.9.77.131:35103)
I20260810 01:46:47.045296 12919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:47.041248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:55047 (local address 127.9.77.129:39511)
0810 01:46:47.041422 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:47.041428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:47.041683 (+ 255us) server_negotiation.cc:407] Connection header received
0810 01:46:47.041893 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:47.041899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:47.041959 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:47.042055 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:46:47.042380 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.042963 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.044155 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.044621 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.044805 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:47.044878 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:47.045051 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 01:46:47.045161 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 01:46:47.045322 12958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:47.041069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:39511 (local address 127.9.77.130:55047)
0810 01:46:47.041560 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:47.041573 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:47.041763 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:47.042070 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:47.042073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:47.042089 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:46:47.042259 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:47.042265 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.043086 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.043091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:47.043991 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:47.044000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.044780 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.044784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:47.044894 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:47.044928 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 01:46:47.045032 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 01:46:47.045116 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":251,"spinlock_wait_cycles":14976,"thread_start_us":99,"threads_started":1}
I20260810 01:46:47.045782 12923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:47.041228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:49507 (local address 127.9.77.131:35103)
0810 01:46:47.041414 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:47.041420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:47.041673 (+ 253us) server_negotiation.cc:407] Connection header received
0810 01:46:47.041925 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:47.041928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:47.041981 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:47.042073 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:46:47.042887 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.043446 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.044498 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.045071 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.045211 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:47.045272 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:47.045497 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 01:46:47.045599 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 01:46:47.046031 12400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f27eccf1a494667bcbe2a6c28abac4c" candidate_uuid: "9e29676918e746de84c4e8b62f658f35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40a6de2a6f85463998b46a214b210bb2" is_pre_election: true
I20260810 01:46:47.046262 12957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:47.041069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:35103 (local address 127.9.77.130:49507)
0810 01:46:47.041542 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:47.041567 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:47.041757 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:47.042085 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:47.042089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:47.042105 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:46:47.042754 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:47.042763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.043568 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.043573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:47.044367 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:47.044376 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.045201 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.045205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:47.045309 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:47.045340 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 01:46:47.045977 (+ 637us) client_negotiation.cc:240] Negotiation successful
0810 01:46:47.046059 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":1,"spinlock_wait_cycles":40064,"thread_start_us":141,"threads_started":1}
I20260810 01:46:47.046236 12400 raft_consensus.cc:2454] T 1f27eccf1a494667bcbe2a6c28abac4c P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e29676918e746de84c4e8b62f658f35 in term 0.
I20260810 01:46:47.046693 12511 leader_election.cc:302] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40a6de2a6f85463998b46a214b210bb2, 9e29676918e746de84c4e8b62f658f35; no voters:
I20260810 01:46:47.046962 12819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f27eccf1a494667bcbe2a6c28abac4c" candidate_uuid: "9e29676918e746de84c4e8b62f658f35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" is_pre_election: true
I20260810 01:46:47.047101 12935 raft_consensus.cc:2781] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:46:47.047174 12935 raft_consensus.cc:482] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:46:47.047216 12935 raft_consensus.cc:3037] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:47.047255 12819 raft_consensus.cc:2454] T 1f27eccf1a494667bcbe2a6c28abac4c P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e29676918e746de84c4e8b62f658f35 in term 0.
I20260810 01:46:47.048621 12935 raft_consensus.cc:504] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:47.048959 12935 leader_election.cc:288] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35 [CANDIDATE]: Term 1 election: Requested vote from peers 40a6de2a6f85463998b46a214b210bb2 (127.9.77.129:39511), e7a50c0229de40a6bc28bd32ea6649ef (127.9.77.131:35103)
I20260810 01:46:47.049382 12400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f27eccf1a494667bcbe2a6c28abac4c" candidate_uuid: "9e29676918e746de84c4e8b62f658f35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40a6de2a6f85463998b46a214b210bb2"
I20260810 01:46:47.049366 12819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f27eccf1a494667bcbe2a6c28abac4c" candidate_uuid: "9e29676918e746de84c4e8b62f658f35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a50c0229de40a6bc28bd32ea6649ef"
I20260810 01:46:47.049518 12819 raft_consensus.cc:3037] T 1f27eccf1a494667bcbe2a6c28abac4c P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:47.049520 12400 raft_consensus.cc:3037] T 1f27eccf1a494667bcbe2a6c28abac4c P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:47.050873 12819 raft_consensus.cc:2454] T 1f27eccf1a494667bcbe2a6c28abac4c P e7a50c0229de40a6bc28bd32ea6649ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e29676918e746de84c4e8b62f658f35 in term 1.
I20260810 01:46:47.050870 12400 raft_consensus.cc:2454] T 1f27eccf1a494667bcbe2a6c28abac4c P 40a6de2a6f85463998b46a214b210bb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e29676918e746de84c4e8b62f658f35 in term 1.
I20260810 01:46:47.051152 12511 leader_election.cc:302] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40a6de2a6f85463998b46a214b210bb2, 9e29676918e746de84c4e8b62f658f35; no voters:
I20260810 01:46:47.051334 12935 raft_consensus.cc:2781] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:46:47.051554 12935 raft_consensus.cc:686] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35 [term 1 LEADER]: Becoming Leader. State: Replica: 9e29676918e746de84c4e8b62f658f35, State: Running, Role: LEADER
I20260810 01:46:47.051962 12935 consensus_queue.cc:237] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:47.054973 12959 tablet_replica.cc:439] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35: TxnStatusTablet state changed. Reason: New leader 9e29676918e746de84c4e8b62f658f35. Latest consensus state: current_term: 1 leader_uuid: "9e29676918e746de84c4e8b62f658f35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } } }
I20260810 01:46:47.055094 12959 tablet_replica.cc:442] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35: This TxnStatusTablet replica's current role is: LEADER
I20260810 01:46:47.055413 12964 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 01:46:47.055701 12964 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 01:46:47.056484 12163 catalog_manager.cc:5495] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e29676918e746de84c4e8b62f658f35 (127.9.77.130). New cstate: current_term: 1 leader_uuid: "9e29676918e746de84c4e8b62f658f35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } health_report { overall_health: UNKNOWN } } }
I20260810 01:46:47.079489 12950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:47.074621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:45423 (local address 127.0.0.1:33376)
0810 01:46:47.074845 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:47.074870 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:47.075034 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:47.075458 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:47.075464 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:47.075474 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 01:46:47.075623 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:47.075629 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.076866 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.076869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:47.077535 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:47.077543 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.078138 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.078141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:47.078217 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:47.078271 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 01:46:47.079175 (+ 904us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 01:46:47.079188 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 01:46:47.079243 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 01:46:47.079330 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 01:46:47.079511 12924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:47.074756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33376 (local address 127.9.77.130:45423)
0810 01:46:47.074935 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:47.074941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:47.074960 (+ 19us) server_negotiation.cc:407] Connection header received
0810 01:46:47.075217 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:47.075220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:47.075297 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:47.075403 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:46:47.075751 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.076659 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.077657 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.078020 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.078131 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:47.078425 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:46:47.079082 (+ 657us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:46:47.079202 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:47.079259 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 01:46:47.079333 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 01:46:47.086727 12400 raft_consensus.cc:1261] T 1f27eccf1a494667bcbe2a6c28abac4c P 40a6de2a6f85463998b46a214b210bb2 [term 1 FOLLOWER]: Refusing update from remote peer 9e29676918e746de84c4e8b62f658f35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:46:47.087596 12819 raft_consensus.cc:1261] T 1f27eccf1a494667bcbe2a6c28abac4c P e7a50c0229de40a6bc28bd32ea6649ef [term 1 FOLLOWER]: Refusing update from remote peer 9e29676918e746de84c4e8b62f658f35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:46:47.087580 12935 consensus_queue.cc:1035] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:46:47.088406 12935 consensus_queue.cc:1035] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:46:47.093055 12933 tablet_replica.cc:439] T 1f27eccf1a494667bcbe2a6c28abac4c P e7a50c0229de40a6bc28bd32ea6649ef: TxnStatusTablet state changed. Reason: New leader 9e29676918e746de84c4e8b62f658f35. Latest consensus state: current_term: 1 leader_uuid: "9e29676918e746de84c4e8b62f658f35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } } }
I20260810 01:46:47.093222 12933 tablet_replica.cc:442] T 1f27eccf1a494667bcbe2a6c28abac4c P e7a50c0229de40a6bc28bd32ea6649ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 01:46:47.093532 12936 tablet_replica.cc:439] T 1f27eccf1a494667bcbe2a6c28abac4c P 40a6de2a6f85463998b46a214b210bb2: TxnStatusTablet state changed. Reason: New leader 9e29676918e746de84c4e8b62f658f35. Latest consensus state: current_term: 1 leader_uuid: "9e29676918e746de84c4e8b62f658f35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } } }
I20260810 01:46:47.093688 12936 tablet_replica.cc:442] T 1f27eccf1a494667bcbe2a6c28abac4c P 40a6de2a6f85463998b46a214b210bb2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 01:46:47.097993 12959 tablet_replica.cc:439] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9e29676918e746de84c4e8b62f658f35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } } }
I20260810 01:46:47.098152 12959 tablet_replica.cc:442] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35: This TxnStatusTablet replica's current role is: LEADER
I20260810 01:46:47.098685 12968 mvcc.cc:205] Tried to move back new op lower bound from 7316792963418238976 to 7316792963288469504. Current Snapshot: MvccSnapshot[applied={T|T < 7316792963418238976}]
I20260810 01:46:47.098671 12933 tablet_replica.cc:439] T 1f27eccf1a494667bcbe2a6c28abac4c P e7a50c0229de40a6bc28bd32ea6649ef: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9e29676918e746de84c4e8b62f658f35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } } }
I20260810 01:46:47.098799 12933 tablet_replica.cc:442] T 1f27eccf1a494667bcbe2a6c28abac4c P e7a50c0229de40a6bc28bd32ea6649ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 01:46:47.101519 12971 mvcc.cc:205] Tried to move back new op lower bound from 7316792963418238976 to 7316792963288469504. Current Snapshot: MvccSnapshot[applied={T|T < 7316792963418238976}]
I20260810 01:46:47.105561 12959 tablet_replica.cc:439] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9e29676918e746de84c4e8b62f658f35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } } }
I20260810 01:46:47.105705 12959 tablet_replica.cc:442] T 1f27eccf1a494667bcbe2a6c28abac4c P 9e29676918e746de84c4e8b62f658f35: This TxnStatusTablet replica's current role is: LEADER
I20260810 01:46:47.107096 12936 tablet_replica.cc:439] T 1f27eccf1a494667bcbe2a6c28abac4c P 40a6de2a6f85463998b46a214b210bb2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9e29676918e746de84c4e8b62f658f35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } } }
I20260810 01:46:47.107232 12936 tablet_replica.cc:442] T 1f27eccf1a494667bcbe2a6c28abac4c P 40a6de2a6f85463998b46a214b210bb2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 01:46:47.114764 12923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:47.110643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54424 (local address 127.9.77.131:35103)
0810 01:46:47.110767 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:47.110772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:47.110888 (+ 116us) server_negotiation.cc:407] Connection header received
0810 01:46:47.111080 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:47.111083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:47.111154 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:47.111270 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:46:47.111560 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.112213 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.113126 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.113379 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.113493 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:47.113807 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:46:47.114318 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:46:47.114455 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:47.114520 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 01:46:47.114588 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 01:46:47.127728 12924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:47.124140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:59117 (local address 127.9.77.130:45423)
0810 01:46:47.124285 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:47.124289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:47.124306 (+ 17us) server_negotiation.cc:407] Connection header received
0810 01:46:47.124373 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:47.124375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:47.124428 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:47.124498 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:46:47.124794 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.125340 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.126609 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.126823 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.126898 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:47.127268 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:46:47.127345 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:46:47.127397 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:47.127538 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 01:46:47.127589 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 01:46:47.127758 12899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:47.123963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:45423 (local address 127.9.77.131:59117)
0810 01:46:47.124171 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:47.124191 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:47.124292 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:47.124536 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:47.124539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:47.124548 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 01:46:47.124688 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:47.124694 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.125444 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.125448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:47.126489 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:47.126500 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.126928 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.126931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:47.127036 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:47.127095 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 01:46:47.127425 (+ 330us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 01:46:47.127435 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 01:46:47.127516 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 01:46:47.127607 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 01:46:47.136936 12611 raft_consensus.cc:1261] T 4818d04f41f043648a20694827dd362f P 9e29676918e746de84c4e8b62f658f35 [term 1 FOLLOWER]: Refusing update from remote peer e7a50c0229de40a6bc28bd32ea6649ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:46:47.137552 12400 raft_consensus.cc:1261] T 4818d04f41f043648a20694827dd362f P 40a6de2a6f85463998b46a214b210bb2 [term 1 FOLLOWER]: Refusing update from remote peer e7a50c0229de40a6bc28bd32ea6649ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:46:47.137703 12938 consensus_queue.cc:1035] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:46:47.137998 12933 consensus_queue.cc:1035] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:46:47.146106 12970 mvcc.cc:205] Tried to move back new op lower bound from 7316792963626782720 to 7316792962537066496. Current Snapshot: MvccSnapshot[applied={T|T < 7316792963626782720}]
I20260810 01:46:47.166709 12985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:47.162074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:35103 (local address 127.9.77.130:38633)
0810 01:46:47.162500 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:47.162519 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:47.162653 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:47.162961 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:47.162964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:47.162972 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 01:46:47.163110 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:47.163122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.163948 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.163953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:47.164918 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:47.164930 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.165627 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.165629 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:47.165750 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:47.165817 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 01:46:47.166222 (+ 405us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 01:46:47.166230 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 01:46:47.166296 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 01:46:47.166503 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":165,"threads_started":1}
I20260810 01:46:47.166689 12923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:47.162066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:38633 (local address 127.9.77.131:35103)
0810 01:46:47.162266 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:47.162273 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:47.162598 (+ 325us) server_negotiation.cc:407] Connection header received
0810 01:46:47.162784 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:47.162789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:47.162859 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:47.162937 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:46:47.163232 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.163812 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.165071 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.165442 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.165526 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:47.165989 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:46:47.166102 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:46:47.166151 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:47.166314 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 01:46:47.166503 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 01:46:47.199466 12166 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:46:47.199766 12166 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:46:47.203469 12166 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad82f728b5a5494a9f9deaf9c241c54f: Sending DeleteTablet for 3 replicas of tablet 4818d04f41f043648a20694827dd362f
I20260810 01:46:47.204540 12375 tablet_service.cc:1514] Processing DeleteTablet for tablet 4818d04f41f043648a20694827dd362f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:46:47 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:35166
I20260810 01:46:47.204843 12583 tablet_service.cc:1514] Processing DeleteTablet for tablet 4818d04f41f043648a20694827dd362f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:46:47 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:33368
I20260810 01:46:47.205713 12786 tablet_service.cc:1514] Processing DeleteTablet for tablet 4818d04f41f043648a20694827dd362f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:46:47 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:54418
I20260810 01:46:47.206823 12989 tablet_replica.cc:330] T 4818d04f41f043648a20694827dd362f P 40a6de2a6f85463998b46a214b210bb2: stopping tablet replica
I20260810 01:46:47.207010 12166 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 01:46:47.207700 12166 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:46:47.207957 12989 raft_consensus.cc:2229] T 4818d04f41f043648a20694827dd362f P 40a6de2a6f85463998b46a214b210bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:46:47.208395 12989 raft_consensus.cc:2258] T 4818d04f41f043648a20694827dd362f P 40a6de2a6f85463998b46a214b210bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:46:47.211035 12989 ts_tablet_manager.cc:1905] T 4818d04f41f043648a20694827dd362f P 40a6de2a6f85463998b46a214b210bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:46:47.213521 12375 tablet_service.cc:1467] Processing CreateTablet for tablet 35aba140b976421781ff2f8929f16cd5 (DEFAULT_TABLE table=test-table [id=9c36b11b599d4eac9c72073b3ec99f03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:46:47.214354 12583 tablet_service.cc:1467] Processing CreateTablet for tablet 35aba140b976421781ff2f8929f16cd5 (DEFAULT_TABLE table=test-table [id=9c36b11b599d4eac9c72073b3ec99f03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:46:47.214411 12786 tablet_service.cc:1467] Processing CreateTablet for tablet 35aba140b976421781ff2f8929f16cd5 (DEFAULT_TABLE table=test-table [id=9c36b11b599d4eac9c72073b3ec99f03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:46:47.214706 12583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35aba140b976421781ff2f8929f16cd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:47.214735 12786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35aba140b976421781ff2f8929f16cd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:47.215955 12996 tablet_replica.cc:330] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef: stopping tablet replica
I20260810 01:46:47.218107 12928 tablet_bootstrap.cc:492] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35: Bootstrap starting.
I20260810 01:46:47.218868 12375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35aba140b976421781ff2f8929f16cd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:47.219251 12928 tablet_bootstrap.cc:654] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:47.221727 12926 tablet_bootstrap.cc:492] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2: Bootstrap starting.
I20260810 01:46:47.222831 12926 tablet_bootstrap.cc:654] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:47.223366 12989 ts_tablet_manager.cc:1918] T 4818d04f41f043648a20694827dd362f P 40a6de2a6f85463998b46a214b210bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 01:46:47.223498 12989 log.cc:1196] T 4818d04f41f043648a20694827dd362f P 40a6de2a6f85463998b46a214b210bb2: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-0/wal/wals/4818d04f41f043648a20694827dd362f
I20260810 01:46:47.223922 12989 ts_tablet_manager.cc:1939] T 4818d04f41f043648a20694827dd362f P 40a6de2a6f85463998b46a214b210bb2: Deleting consensus metadata
I20260810 01:46:47.218150 12927 tablet_bootstrap.cc:492] T 35aba140b976421781ff2f8929f16cd5 P e7a50c0229de40a6bc28bd32ea6649ef: Bootstrap starting.
I20260810 01:46:47.224953 12927 tablet_bootstrap.cc:654] T 35aba140b976421781ff2f8929f16cd5 P e7a50c0229de40a6bc28bd32ea6649ef: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:47.209617 12995 tablet_replica.cc:330] T 4818d04f41f043648a20694827dd362f P 9e29676918e746de84c4e8b62f658f35: stopping tablet replica
I20260810 01:46:47.225879 12139 catalog_manager.cc:4841] TS 40a6de2a6f85463998b46a214b210bb2 (127.9.77.129:39511): tablet 4818d04f41f043648a20694827dd362f (table test-table [id=03aaa2032725434498d755e565a98b7a]) successfully deleted
I20260810 01:46:47.225946 12996 raft_consensus.cc:2229] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef [term 1 LEADER]: Raft consensus shutting down.
I20260810 01:46:47.225939 12995 raft_consensus.cc:2229] T 4818d04f41f043648a20694827dd362f P 9e29676918e746de84c4e8b62f658f35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:46:47.226145 12995 raft_consensus.cc:2258] T 4818d04f41f043648a20694827dd362f P 9e29676918e746de84c4e8b62f658f35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:46:47.226416 12996 raft_consensus.cc:2258] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:46:47.227797 12996 ts_tablet_manager.cc:1905] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:46:47.228689 12927 tablet_bootstrap.cc:492] T 35aba140b976421781ff2f8929f16cd5 P e7a50c0229de40a6bc28bd32ea6649ef: No bootstrap required, opened a new log
I20260810 01:46:47.228781 12927 ts_tablet_manager.cc:1397] T 35aba140b976421781ff2f8929f16cd5 P e7a50c0229de40a6bc28bd32ea6649ef: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260810 01:46:47.229028 12995 ts_tablet_manager.cc:1905] T 4818d04f41f043648a20694827dd362f P 9e29676918e746de84c4e8b62f658f35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:46:47.229218 12926 tablet_bootstrap.cc:492] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2: No bootstrap required, opened a new log
I20260810 01:46:47.229306 12926 ts_tablet_manager.cc:1397] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260810 01:46:47.229369 12927 raft_consensus.cc:348] T 35aba140b976421781ff2f8929f16cd5 P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:47.229508 12927 raft_consensus.cc:374] T 35aba140b976421781ff2f8929f16cd5 P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:47.229542 12927 raft_consensus.cc:729] T 35aba140b976421781ff2f8929f16cd5 P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a50c0229de40a6bc28bd32ea6649ef, State: Initialized, Role: FOLLOWER
I20260810 01:46:47.229704 12927 consensus_queue.cc:260] T 35aba140b976421781ff2f8929f16cd5 P e7a50c0229de40a6bc28bd32ea6649ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:47.229842 12926 raft_consensus.cc:348] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:47.229954 12926 raft_consensus.cc:374] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:47.229986 12926 raft_consensus.cc:729] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40a6de2a6f85463998b46a214b210bb2, State: Initialized, Role: FOLLOWER
I20260810 01:46:47.230033 12927 ts_tablet_manager.cc:1428] T 35aba140b976421781ff2f8929f16cd5 P e7a50c0229de40a6bc28bd32ea6649ef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 01:46:47.230109 12926 consensus_queue.cc:260] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:47.230401 12926 ts_tablet_manager.cc:1428] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 01:46:47.232439 12995 ts_tablet_manager.cc:1918] T 4818d04f41f043648a20694827dd362f P 9e29676918e746de84c4e8b62f658f35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 01:46:47.232543 12995 log.cc:1196] T 4818d04f41f043648a20694827dd362f P 9e29676918e746de84c4e8b62f658f35: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-1/wal/wals/4818d04f41f043648a20694827dd362f
I20260810 01:46:47.232883 12995 ts_tablet_manager.cc:1939] T 4818d04f41f043648a20694827dd362f P 9e29676918e746de84c4e8b62f658f35: Deleting consensus metadata
I20260810 01:46:47.234304 12143 catalog_manager.cc:4841] TS 9e29676918e746de84c4e8b62f658f35 (127.9.77.130:45423): tablet 4818d04f41f043648a20694827dd362f (table test-table [id=03aaa2032725434498d755e565a98b7a]) successfully deleted
I20260810 01:46:47.234884 12996 ts_tablet_manager.cc:1918] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 01:46:47.234998 12996 log.cc:1196] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-2/wal/wals/4818d04f41f043648a20694827dd362f
I20260810 01:46:47.235116 12928 tablet_bootstrap.cc:492] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35: No bootstrap required, opened a new log
I20260810 01:46:47.235208 12928 ts_tablet_manager.cc:1397] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20260810 01:46:47.235361 12996 ts_tablet_manager.cc:1939] T 4818d04f41f043648a20694827dd362f P e7a50c0229de40a6bc28bd32ea6649ef: Deleting consensus metadata
I20260810 01:46:47.235765 12928 raft_consensus.cc:348] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:47.235890 12928 raft_consensus.cc:374] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:47.235924 12928 raft_consensus.cc:729] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e29676918e746de84c4e8b62f658f35, State: Initialized, Role: FOLLOWER
I20260810 01:46:47.236617 12141 catalog_manager.cc:4841] TS e7a50c0229de40a6bc28bd32ea6649ef (127.9.77.131:35103): tablet 4818d04f41f043648a20694827dd362f (table test-table [id=03aaa2032725434498d755e565a98b7a]) successfully deleted
I20260810 01:46:47.236644 12928 consensus_queue.cc:260] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:47.237581 12928 ts_tablet_manager.cc:1428] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 01:46:47.337649 12936 raft_consensus.cc:482] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:46:47.337895 12936 raft_consensus.cc:504] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:47.339723 12936 leader_election.cc:288] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e29676918e746de84c4e8b62f658f35 (127.9.77.130:45423), e7a50c0229de40a6bc28bd32ea6649ef (127.9.77.131:35103)
I20260810 01:46:47.344852 12923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:47.340288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:38297 (local address 127.9.77.131:35103)
0810 01:46:47.340607 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:47.340613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:47.340631 (+ 18us) server_negotiation.cc:407] Connection header received
0810 01:46:47.340772 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:47.340775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:47.340831 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:47.340929 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:46:47.341339 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.342195 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.343500 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.344034 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.344177 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:47.344244 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:47.344609 (+ 365us) server_negotiation.cc:299] Negotiation successful
0810 01:46:47.344707 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
I20260810 01:46:47.344858 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:47.340047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:35103 (local address 127.9.77.129:38297)
0810 01:46:47.340476 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:47.340493 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:47.340630 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:47.340988 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:47.340991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:47.341012 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:46:47.341172 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:47.341179 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.342331 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.342334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:47.343366 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:47.343376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.344339 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.344343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:47.344448 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:47.344475 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 01:46:47.344560 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 01:46:47.344650 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":75,"thread_start_us":120,"threads_started":1}
I20260810 01:46:47.345415 12924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:47.340235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:56265 (local address 127.9.77.130:45423)
0810 01:46:47.340481 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:47.340489 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:47.340919 (+ 430us) server_negotiation.cc:407] Connection header received
0810 01:46:47.341140 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:47.341145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:47.341206 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:47.341289 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:46:47.341645 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.342332 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.343856 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.344541 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.344652 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:47.344727 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:47.345166 (+ 439us) server_negotiation.cc:299] Negotiation successful
0810 01:46:47.345249 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260810 01:46:47.345685 12819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35aba140b976421781ff2f8929f16cd5" candidate_uuid: "40a6de2a6f85463998b46a214b210bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" is_pre_election: true
I20260810 01:46:47.345901 12819 raft_consensus.cc:2454] T 35aba140b976421781ff2f8929f16cd5 P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40a6de2a6f85463998b46a214b210bb2 in term 0.
I20260810 01:46:47.345949 13002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:47.340047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:45423 (local address 127.9.77.129:56265)
0810 01:46:47.340786 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:47.340803 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:47.340946 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:47.341338 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:47.341341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:47.341357 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:46:47.341486 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:47.341511 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.342456 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.342460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:47.343719 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:47.343731 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.344869 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.344875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:47.344985 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:47.345035 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 01:46:47.345573 (+ 538us) client_negotiation.cc:240] Negotiation successful
0810 01:46:47.345643 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":257,"thread_start_us":130,"threads_started":1}
I20260810 01:46:47.345991 12959 raft_consensus.cc:482] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:46:47.346107 12959 raft_consensus.cc:504] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:47.346586 12959 leader_election.cc:288] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40a6de2a6f85463998b46a214b210bb2 (127.9.77.129:39511), e7a50c0229de40a6bc28bd32ea6649ef (127.9.77.131:35103)
I20260810 01:46:47.346709 12611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35aba140b976421781ff2f8929f16cd5" candidate_uuid: "40a6de2a6f85463998b46a214b210bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e29676918e746de84c4e8b62f658f35" is_pre_election: true
I20260810 01:46:47.346868 12611 raft_consensus.cc:2454] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40a6de2a6f85463998b46a214b210bb2 in term 0.
I20260810 01:46:47.346912 12301 leader_election.cc:302] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40a6de2a6f85463998b46a214b210bb2, e7a50c0229de40a6bc28bd32ea6649ef; no voters:
I20260810 01:46:47.347316 12936 raft_consensus.cc:2781] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:46:47.347393 12936 raft_consensus.cc:482] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:46:47.347362 12819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35aba140b976421781ff2f8929f16cd5" candidate_uuid: "9e29676918e746de84c4e8b62f658f35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" is_pre_election: true
I20260810 01:46:47.347433 12936 raft_consensus.cc:3037] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:47.347525 12819 raft_consensus.cc:2454] T 35aba140b976421781ff2f8929f16cd5 P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e29676918e746de84c4e8b62f658f35 in term 0.
I20260810 01:46:47.347977 12513 leader_election.cc:302] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e29676918e746de84c4e8b62f658f35, e7a50c0229de40a6bc28bd32ea6649ef; no voters:
I20260810 01:46:47.348052 12400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35aba140b976421781ff2f8929f16cd5" candidate_uuid: "9e29676918e746de84c4e8b62f658f35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40a6de2a6f85463998b46a214b210bb2" is_pre_election: true
I20260810 01:46:47.348405 12959 raft_consensus.cc:2781] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:46:47.348482 12959 raft_consensus.cc:482] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:46:47.348553 12959 raft_consensus.cc:3037] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:47.349186 12936 raft_consensus.cc:504] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:47.349591 12936 leader_election.cc:288] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 9e29676918e746de84c4e8b62f658f35 (127.9.77.130:45423), e7a50c0229de40a6bc28bd32ea6649ef (127.9.77.131:35103)
I20260810 01:46:47.349766 12400 raft_consensus.cc:2379] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e29676918e746de84c4e8b62f658f35 in current term 1: Already voted for candidate 40a6de2a6f85463998b46a214b210bb2 in this term.
I20260810 01:46:47.349872 12959 raft_consensus.cc:504] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:47.350276 12959 leader_election.cc:288] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35 [CANDIDATE]: Term 1 election: Requested vote from peers 40a6de2a6f85463998b46a214b210bb2 (127.9.77.129:39511), e7a50c0229de40a6bc28bd32ea6649ef (127.9.77.131:35103)
I20260810 01:46:47.350556 12400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35aba140b976421781ff2f8929f16cd5" candidate_uuid: "9e29676918e746de84c4e8b62f658f35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40a6de2a6f85463998b46a214b210bb2"
I20260810 01:46:47.350698 12400 raft_consensus.cc:2379] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e29676918e746de84c4e8b62f658f35 in current term 1: Already voted for candidate 40a6de2a6f85463998b46a214b210bb2 in this term.
I20260810 01:46:47.350697 12819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35aba140b976421781ff2f8929f16cd5" candidate_uuid: "9e29676918e746de84c4e8b62f658f35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a50c0229de40a6bc28bd32ea6649ef"
I20260810 01:46:47.350816 12818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35aba140b976421781ff2f8929f16cd5" candidate_uuid: "40a6de2a6f85463998b46a214b210bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a50c0229de40a6bc28bd32ea6649ef"
I20260810 01:46:47.350847 12819 raft_consensus.cc:3037] T 35aba140b976421781ff2f8929f16cd5 P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:47.351410 12611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35aba140b976421781ff2f8929f16cd5" candidate_uuid: "40a6de2a6f85463998b46a214b210bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e29676918e746de84c4e8b62f658f35"
I20260810 01:46:47.351572 12611 raft_consensus.cc:2379] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 40a6de2a6f85463998b46a214b210bb2 in current term 1: Already voted for candidate 9e29676918e746de84c4e8b62f658f35 in this term.
I20260810 01:46:47.351886 12303 leader_election.cc:302] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40a6de2a6f85463998b46a214b210bb2; no voters: 9e29676918e746de84c4e8b62f658f35, e7a50c0229de40a6bc28bd32ea6649ef
I20260810 01:46:47.352121 12936 raft_consensus.cc:2726] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 01:46:47.352365 12819 raft_consensus.cc:2454] T 35aba140b976421781ff2f8929f16cd5 P e7a50c0229de40a6bc28bd32ea6649ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e29676918e746de84c4e8b62f658f35 in term 1.
I20260810 01:46:47.352728 12513 leader_election.cc:302] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9e29676918e746de84c4e8b62f658f35, e7a50c0229de40a6bc28bd32ea6649ef; no voters: 40a6de2a6f85463998b46a214b210bb2
I20260810 01:46:47.352907 12959 raft_consensus.cc:2781] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:46:47.353001 12959 raft_consensus.cc:686] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35 [term 1 LEADER]: Becoming Leader. State: Replica: 9e29676918e746de84c4e8b62f658f35, State: Running, Role: LEADER
I20260810 01:46:47.353166 12959 consensus_queue.cc:237] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:47.355262 12166 catalog_manager.cc:5495] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e29676918e746de84c4e8b62f658f35 (127.9.77.130). New cstate: current_term: 1 leader_uuid: "9e29676918e746de84c4e8b62f658f35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } health_report { overall_health: UNKNOWN } } }
I20260810 01:46:47.398456 12924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:47.395212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33382 (local address 127.9.77.130:45423)
0810 01:46:47.395345 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:47.395352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:47.395372 (+ 20us) server_negotiation.cc:407] Connection header received
0810 01:46:47.395480 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:47.395483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:47.395532 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:47.395600 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:46:47.395874 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.396513 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.397434 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.397660 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.397740 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:47.398010 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:46:47.398091 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:46:47.398146 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:47.398259 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 01:46:47.398314 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 01:46:47.408776 12924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:47.405406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:34439 (local address 127.9.77.130:45423)
0810 01:46:47.405512 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:47.405515 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:47.405560 (+ 45us) server_negotiation.cc:407] Connection header received
0810 01:46:47.405725 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:47.405727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:47.405768 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:47.405846 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:46:47.406121 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.406683 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.407694 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.407914 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.407995 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:47.408252 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:46:47.408321 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:46:47.408375 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:47.408513 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 01:46:47.408558 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 01:46:47.408776 12985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:47.405352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:45423 (local address 127.9.77.130:34439)
0810 01:46:47.405501 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:47.405522 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:47.405597 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:47.405866 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:47.405868 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:47.405873 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 01:46:47.406013 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:47.406017 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.406802 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.406805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:47.407554 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:47.407564 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:47.408011 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:47.408051 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:47.408129 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:47.408155 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 01:46:47.408397 (+ 242us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 01:46:47.408402 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 01:46:47.408490 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 01:46:47.408573 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 01:46:47.415321 12819 raft_consensus.cc:1261] T 35aba140b976421781ff2f8929f16cd5 P e7a50c0229de40a6bc28bd32ea6649ef [term 1 FOLLOWER]: Refusing update from remote peer 9e29676918e746de84c4e8b62f658f35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:46:47.415321 12400 raft_consensus.cc:1261] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2 [term 1 FOLLOWER]: Refusing update from remote peer 9e29676918e746de84c4e8b62f658f35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:46:47.415745 12935 consensus_queue.cc:1035] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:46:47.415910 12959 consensus_queue.cc:1035] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 01:46:47.431113 12968 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 01:46:48.995018 12166 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:46:48.995236 12166 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:46:48.998170 12166 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad82f728b5a5494a9f9deaf9c241c54f: Sending DeleteTablet for 3 replicas of tablet 35aba140b976421781ff2f8929f16cd5
I20260810 01:46:48.998855 12582 tablet_service.cc:1514] Processing DeleteTablet for tablet 35aba140b976421781ff2f8929f16cd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:46:48 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:33368
I20260810 01:46:48.998853 12375 tablet_service.cc:1514] Processing DeleteTablet for tablet 35aba140b976421781ff2f8929f16cd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:46:48 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:35166
I20260810 01:46:48.999151 12786 tablet_service.cc:1514] Processing DeleteTablet for tablet 35aba140b976421781ff2f8929f16cd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:46:48 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:54418
I20260810 01:46:48.999284 13043 tablet_replica.cc:330] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35: stopping tablet replica
I20260810 01:46:48.999521 13043 raft_consensus.cc:2229] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35 [term 1 LEADER]: Raft consensus shutting down.
I20260810 01:46:48.999622 13045 tablet_replica.cc:330] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2: stopping tablet replica
I20260810 01:46:48.999760 13045 raft_consensus.cc:2229] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:46:48.999773 13046 tablet_replica.cc:330] T 35aba140b976421781ff2f8929f16cd5 P e7a50c0229de40a6bc28bd32ea6649ef: stopping tablet replica
I20260810 01:46:48.999892 13045 raft_consensus.cc:2258] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:46:48.999894 13043 raft_consensus.cc:2258] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:46:49.000151 13046 raft_consensus.cc:2229] T 35aba140b976421781ff2f8929f16cd5 P e7a50c0229de40a6bc28bd32ea6649ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:46:49.000353 13046 raft_consensus.cc:2258] T 35aba140b976421781ff2f8929f16cd5 P e7a50c0229de40a6bc28bd32ea6649ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:46:49.000785 13045 ts_tablet_manager.cc:1905] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:46:49.003983 13043 ts_tablet_manager.cc:1905] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:46:49.004449 13045 ts_tablet_manager.cc:1918] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 01:46:49.004542 13045 log.cc:1196] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-0/wal/wals/35aba140b976421781ff2f8929f16cd5
I20260810 01:46:49.004851 13045 ts_tablet_manager.cc:1939] T 35aba140b976421781ff2f8929f16cd5 P 40a6de2a6f85463998b46a214b210bb2: Deleting consensus metadata
I20260810 01:46:49.005702 12139 catalog_manager.cc:4841] TS 40a6de2a6f85463998b46a214b210bb2 (127.9.77.129:39511): tablet 35aba140b976421781ff2f8929f16cd5 (table test-table [id=9c36b11b599d4eac9c72073b3ec99f03]) successfully deleted
I20260810 01:46:49.006666 13046 ts_tablet_manager.cc:1905] T 35aba140b976421781ff2f8929f16cd5 P e7a50c0229de40a6bc28bd32ea6649ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:46:49.007392 13043 ts_tablet_manager.cc:1918] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 01:46:49.007509 13043 log.cc:1196] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-1/wal/wals/35aba140b976421781ff2f8929f16cd5
I20260810 01:46:49.007862 13043 ts_tablet_manager.cc:1939] T 35aba140b976421781ff2f8929f16cd5 P 9e29676918e746de84c4e8b62f658f35: Deleting consensus metadata
I20260810 01:46:49.008745 12143 catalog_manager.cc:4841] TS 9e29676918e746de84c4e8b62f658f35 (127.9.77.130:45423): tablet 35aba140b976421781ff2f8929f16cd5 (table test-table [id=9c36b11b599d4eac9c72073b3ec99f03]) successfully deleted
I20260810 01:46:49.009702 13046 ts_tablet_manager.cc:1918] T 35aba140b976421781ff2f8929f16cd5 P e7a50c0229de40a6bc28bd32ea6649ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 01:46:49.009773 13046 log.cc:1196] T 35aba140b976421781ff2f8929f16cd5 P e7a50c0229de40a6bc28bd32ea6649ef: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-2/wal/wals/35aba140b976421781ff2f8929f16cd5
I20260810 01:46:49.010051 13046 ts_tablet_manager.cc:1939] T 35aba140b976421781ff2f8929f16cd5 P e7a50c0229de40a6bc28bd32ea6649ef: Deleting consensus metadata
I20260810 01:46:49.010842 12141 catalog_manager.cc:4841] TS e7a50c0229de40a6bc28bd32ea6649ef (127.9.77.131:35103): tablet 35aba140b976421781ff2f8929f16cd5 (table test-table [id=9c36b11b599d4eac9c72073b3ec99f03]) successfully deleted
Aug 10 01:46:49 dist-test-slave-kvfs krb5kdc[12097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326409, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 01:46:49 dist-test-slave-kvfs krb5kdc[12097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326409, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:49.043346 13058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:49.027358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41276 (local address 127.9.77.190:46807)
0810 01:46:49.027763 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:49.027774 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:49.027794 (+ 20us) server_negotiation.cc:407] Connection header received
0810 01:46:49.027975 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:49.027978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:49.028101 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:49.028228 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:49.029063 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:49.029788 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:49.030733 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:49.030999 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:49.031117 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:49.039182 (+ 8065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:49.039186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:49.039188 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:49.039225 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:49.041115 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:49.041726 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:49.041733 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:49.041735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:49.041846 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:49.042175 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:49.042179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:49.042181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:49.042458 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:49.042636 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:49.042922 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 01:46:49.043093 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":149,"threads_started":1}
I20260810 01:46:49.047159 12166 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 01:46:49.047811 12166 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:46:49.054024 12582 tablet_service.cc:1467] Processing CreateTablet for tablet 516a6ba7a82a48ceb598840bd2a0464e (DEFAULT_TABLE table=test-table [id=3a8132c84b05467e93ebf061da872034]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:46:49.054049 12375 tablet_service.cc:1467] Processing CreateTablet for tablet 516a6ba7a82a48ceb598840bd2a0464e (DEFAULT_TABLE table=test-table [id=3a8132c84b05467e93ebf061da872034]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:46:49.054028 12786 tablet_service.cc:1467] Processing CreateTablet for tablet 516a6ba7a82a48ceb598840bd2a0464e (DEFAULT_TABLE table=test-table [id=3a8132c84b05467e93ebf061da872034]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:46:49.054440 12786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 516a6ba7a82a48ceb598840bd2a0464e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:49.054438 12375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 516a6ba7a82a48ceb598840bd2a0464e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:49.054435 12582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 516a6ba7a82a48ceb598840bd2a0464e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:49.057214 13059 tablet_bootstrap.cc:492] T 516a6ba7a82a48ceb598840bd2a0464e P 9e29676918e746de84c4e8b62f658f35: Bootstrap starting.
I20260810 01:46:49.057557 13060 tablet_bootstrap.cc:492] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2: Bootstrap starting.
I20260810 01:46:49.057956 13062 tablet_bootstrap.cc:492] T 516a6ba7a82a48ceb598840bd2a0464e P e7a50c0229de40a6bc28bd32ea6649ef: Bootstrap starting.
I20260810 01:46:49.058561 13060 tablet_bootstrap.cc:654] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:49.058571 13059 tablet_bootstrap.cc:654] T 516a6ba7a82a48ceb598840bd2a0464e P 9e29676918e746de84c4e8b62f658f35: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:49.059041 13062 tablet_bootstrap.cc:654] T 516a6ba7a82a48ceb598840bd2a0464e P e7a50c0229de40a6bc28bd32ea6649ef: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:49.059964 13059 tablet_bootstrap.cc:492] T 516a6ba7a82a48ceb598840bd2a0464e P 9e29676918e746de84c4e8b62f658f35: No bootstrap required, opened a new log
I20260810 01:46:49.059964 13060 tablet_bootstrap.cc:492] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2: No bootstrap required, opened a new log
I20260810 01:46:49.060083 13060 ts_tablet_manager.cc:1397] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 01:46:49.060086 13059 ts_tablet_manager.cc:1397] T 516a6ba7a82a48ceb598840bd2a0464e P 9e29676918e746de84c4e8b62f658f35: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 01:46:49.060329 13062 tablet_bootstrap.cc:492] T 516a6ba7a82a48ceb598840bd2a0464e P e7a50c0229de40a6bc28bd32ea6649ef: No bootstrap required, opened a new log
I20260810 01:46:49.060408 13062 ts_tablet_manager.cc:1397] T 516a6ba7a82a48ceb598840bd2a0464e P e7a50c0229de40a6bc28bd32ea6649ef: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 01:46:49.060597 13059 raft_consensus.cc:348] T 516a6ba7a82a48ceb598840bd2a0464e P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:49.060616 13060 raft_consensus.cc:348] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:49.060745 13059 raft_consensus.cc:374] T 516a6ba7a82a48ceb598840bd2a0464e P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:49.060750 13060 raft_consensus.cc:374] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:49.060789 13060 raft_consensus.cc:729] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40a6de2a6f85463998b46a214b210bb2, State: Initialized, Role: FOLLOWER
I20260810 01:46:49.060791 13059 raft_consensus.cc:729] T 516a6ba7a82a48ceb598840bd2a0464e P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e29676918e746de84c4e8b62f658f35, State: Initialized, Role: FOLLOWER
I20260810 01:46:49.060832 13062 raft_consensus.cc:348] T 516a6ba7a82a48ceb598840bd2a0464e P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:49.060946 13062 raft_consensus.cc:374] T 516a6ba7a82a48ceb598840bd2a0464e P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:49.060931 13059 consensus_queue.cc:260] T 516a6ba7a82a48ceb598840bd2a0464e P 9e29676918e746de84c4e8b62f658f35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:49.060932 13060 consensus_queue.cc:260] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:49.060984 13062 raft_consensus.cc:729] T 516a6ba7a82a48ceb598840bd2a0464e P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a50c0229de40a6bc28bd32ea6649ef, State: Initialized, Role: FOLLOWER
I20260810 01:46:49.061131 13062 consensus_queue.cc:260] T 516a6ba7a82a48ceb598840bd2a0464e P e7a50c0229de40a6bc28bd32ea6649ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:49.061266 13059 ts_tablet_manager.cc:1428] T 516a6ba7a82a48ceb598840bd2a0464e P 9e29676918e746de84c4e8b62f658f35: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 01:46:49.061353 13060 ts_tablet_manager.cc:1428] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 01:46:49.061512 13062 ts_tablet_manager.cc:1428] T 516a6ba7a82a48ceb598840bd2a0464e P e7a50c0229de40a6bc28bd32ea6649ef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 01:46:49.171725 13069 raft_consensus.cc:482] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:46:49.171967 13069 raft_consensus.cc:504] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:49.172653 13069 leader_election.cc:288] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e29676918e746de84c4e8b62f658f35 (127.9.77.130:45423), e7a50c0229de40a6bc28bd32ea6649ef (127.9.77.131:35103)
I20260810 01:46:49.173283 12611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516a6ba7a82a48ceb598840bd2a0464e" candidate_uuid: "40a6de2a6f85463998b46a214b210bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e29676918e746de84c4e8b62f658f35" is_pre_election: true
I20260810 01:46:49.173283 12819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516a6ba7a82a48ceb598840bd2a0464e" candidate_uuid: "40a6de2a6f85463998b46a214b210bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" is_pre_election: true
I20260810 01:46:49.173512 12611 raft_consensus.cc:2454] T 516a6ba7a82a48ceb598840bd2a0464e P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40a6de2a6f85463998b46a214b210bb2 in term 0.
I20260810 01:46:49.173525 12819 raft_consensus.cc:2454] T 516a6ba7a82a48ceb598840bd2a0464e P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40a6de2a6f85463998b46a214b210bb2 in term 0.
I20260810 01:46:49.173940 12303 leader_election.cc:302] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40a6de2a6f85463998b46a214b210bb2, 9e29676918e746de84c4e8b62f658f35; no voters:
I20260810 01:46:49.174150 13069 raft_consensus.cc:2781] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:46:49.174208 13069 raft_consensus.cc:482] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:46:49.174237 13069 raft_consensus.cc:3037] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:49.175523 13069 raft_consensus.cc:504] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:49.175902 13069 leader_election.cc:288] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 9e29676918e746de84c4e8b62f658f35 (127.9.77.130:45423), e7a50c0229de40a6bc28bd32ea6649ef (127.9.77.131:35103)
I20260810 01:46:49.176339 12611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516a6ba7a82a48ceb598840bd2a0464e" candidate_uuid: "40a6de2a6f85463998b46a214b210bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e29676918e746de84c4e8b62f658f35"
I20260810 01:46:49.176402 12819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516a6ba7a82a48ceb598840bd2a0464e" candidate_uuid: "40a6de2a6f85463998b46a214b210bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a50c0229de40a6bc28bd32ea6649ef"
I20260810 01:46:49.176506 12611 raft_consensus.cc:3037] T 516a6ba7a82a48ceb598840bd2a0464e P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:49.176561 12819 raft_consensus.cc:3037] T 516a6ba7a82a48ceb598840bd2a0464e P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:49.177713 12611 raft_consensus.cc:2454] T 516a6ba7a82a48ceb598840bd2a0464e P 9e29676918e746de84c4e8b62f658f35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40a6de2a6f85463998b46a214b210bb2 in term 1.
I20260810 01:46:49.177789 12819 raft_consensus.cc:2454] T 516a6ba7a82a48ceb598840bd2a0464e P e7a50c0229de40a6bc28bd32ea6649ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40a6de2a6f85463998b46a214b210bb2 in term 1.
I20260810 01:46:49.178090 12303 leader_election.cc:302] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40a6de2a6f85463998b46a214b210bb2, 9e29676918e746de84c4e8b62f658f35; no voters:
I20260810 01:46:49.178287 13069 raft_consensus.cc:2781] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:46:49.178536 13069 raft_consensus.cc:686] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2 [term 1 LEADER]: Becoming Leader. State: Replica: 40a6de2a6f85463998b46a214b210bb2, State: Running, Role: LEADER
I20260810 01:46:49.178879 13069 consensus_queue.cc:237] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:49.181819 12156 catalog_manager.cc:5495] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40a6de2a6f85463998b46a214b210bb2 (127.9.77.129). New cstate: current_term: 1 leader_uuid: "40a6de2a6f85463998b46a214b210bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } health_report { overall_health: UNKNOWN } } }
I20260810 01:46:49.227873 13074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:49.223346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35180 (local address 127.9.77.129:39511)
0810 01:46:49.223695 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:49.223703 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:49.223715 (+ 12us) server_negotiation.cc:407] Connection header received
0810 01:46:49.223810 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:49.223812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:49.223875 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:49.224006 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:46:49.224330 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:49.224983 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:49.226069 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:49.226331 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:49.226444 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:49.226716 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:46:49.227378 (+ 662us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:46:49.227508 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:49.227558 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 01:46:49.227621 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":179,"threads_started":1}
I20260810 01:46:49.241634 13078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:49.237548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:37899 (local address 127.9.77.130:45423)
0810 01:46:49.237908 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:49.237924 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:49.237943 (+ 19us) server_negotiation.cc:407] Connection header received
0810 01:46:49.238001 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:49.238003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:49.238058 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:49.238169 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:46:49.238487 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:49.239112 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:49.240151 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:49.240423 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:49.240553 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:49.240912 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:46:49.241026 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:46:49.241115 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:49.241318 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 01:46:49.241411 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":126,"threads_started":1}
I20260810 01:46:49.241634 13077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:49.237235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:45423 (local address 127.9.77.129:37899)
0810 01:46:49.237631 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:49.237647 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:49.237784 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:49.238180 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:49.238183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:49.238192 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 01:46:49.238349 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:49.238355 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:49.239239 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:49.239246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:49.239962 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:49.239972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:49.240553 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:49.240557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:49.240678 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:49.240740 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 01:46:49.241119 (+ 379us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 01:46:49.241129 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 01:46:49.241210 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 01:46:49.241345 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":127,"threads_started":1}
I20260810 01:46:49.251910 12611 raft_consensus.cc:1261] T 516a6ba7a82a48ceb598840bd2a0464e P 9e29676918e746de84c4e8b62f658f35 [term 1 FOLLOWER]: Refusing update from remote peer 40a6de2a6f85463998b46a214b210bb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:46:49.252053 12819 raft_consensus.cc:1261] T 516a6ba7a82a48ceb598840bd2a0464e P e7a50c0229de40a6bc28bd32ea6649ef [term 1 FOLLOWER]: Refusing update from remote peer 40a6de2a6f85463998b46a214b210bb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:46:49.252537 13071 consensus_queue.cc:1035] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:46:49.252851 13069 consensus_queue.cc:1035] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:46:49.273808 13074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:49.269796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:43217 (local address 127.9.77.129:39511)
0810 01:46:49.269944 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:49.269948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:49.270201 (+ 253us) server_negotiation.cc:407] Connection header received
0810 01:46:49.270359 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:49.270363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:49.270448 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:49.270551 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:46:49.270831 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:49.271420 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:49.272442 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:49.272689 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:49.272805 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:49.273115 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:46:49.273219 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:46:49.273304 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:49.273553 (+ 249us) server_negotiation.cc:299] Negotiation successful
0810 01:46:49.273622 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 01:46:49.273880 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:49.269752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:39511 (local address 127.9.77.130:43217)
0810 01:46:49.270099 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:49.270116 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:49.270237 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:49.270561 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:49.270564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:49.270572 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 01:46:49.270713 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:49.270722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:49.271538 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:49.271542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:49.272315 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:49.272325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:49.272808 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:49.272812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:49.272917 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:49.272978 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 01:46:49.273375 (+ 397us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 01:46:49.273396 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 01:46:49.273531 (+ 135us) client_negotiation.cc:240] Negotiation successful
0810 01:46:49.273651 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":174,"threads_started":1}
I20260810 01:46:49.300630 12163 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:46:49.300901 12163 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:46:49.303750 12163 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad82f728b5a5494a9f9deaf9c241c54f: Sending DeleteTablet for 3 replicas of tablet 516a6ba7a82a48ceb598840bd2a0464e
I20260810 01:46:49.304553 12375 tablet_service.cc:1514] Processing DeleteTablet for tablet 516a6ba7a82a48ceb598840bd2a0464e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:46:49 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:35166
I20260810 01:46:49.304811 13045 tablet_replica.cc:330] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2: stopping tablet replica
I20260810 01:46:49.304767 12786 tablet_service.cc:1514] Processing DeleteTablet for tablet 516a6ba7a82a48ceb598840bd2a0464e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:46:49 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:54418
I20260810 01:46:49.304963 13045 raft_consensus.cc:2229] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2 [term 1 LEADER]: Raft consensus shutting down.
I20260810 01:46:49.305059 13046 tablet_replica.cc:330] T 516a6ba7a82a48ceb598840bd2a0464e P e7a50c0229de40a6bc28bd32ea6649ef: stopping tablet replica
I20260810 01:46:49.305207 13046 raft_consensus.cc:2229] T 516a6ba7a82a48ceb598840bd2a0464e P e7a50c0229de40a6bc28bd32ea6649ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:46:49.305351 13046 raft_consensus.cc:2258] T 516a6ba7a82a48ceb598840bd2a0464e P e7a50c0229de40a6bc28bd32ea6649ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:46:49.305364 13045 raft_consensus.cc:2258] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:46:49.306010 12163 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 01:46:49.306380 13046 ts_tablet_manager.cc:1905] T 516a6ba7a82a48ceb598840bd2a0464e P e7a50c0229de40a6bc28bd32ea6649ef: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 01:46:49.306589 12163 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:46:49.306614 13045 ts_tablet_manager.cc:1905] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:46:49.310118 13045 ts_tablet_manager.cc:1918] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 01:46:49.310200 13045 log.cc:1196] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-0/wal/wals/516a6ba7a82a48ceb598840bd2a0464e
I20260810 01:46:49.310316 13046 ts_tablet_manager.cc:1918] T 516a6ba7a82a48ceb598840bd2a0464e P e7a50c0229de40a6bc28bd32ea6649ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 01:46:49.310405 13046 log.cc:1196] T 516a6ba7a82a48ceb598840bd2a0464e P e7a50c0229de40a6bc28bd32ea6649ef: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-2/wal/wals/516a6ba7a82a48ceb598840bd2a0464e
I20260810 01:46:49.310457 13045 ts_tablet_manager.cc:1939] T 516a6ba7a82a48ceb598840bd2a0464e P 40a6de2a6f85463998b46a214b210bb2: Deleting consensus metadata
I20260810 01:46:49.310689 13046 ts_tablet_manager.cc:1939] T 516a6ba7a82a48ceb598840bd2a0464e P e7a50c0229de40a6bc28bd32ea6649ef: Deleting consensus metadata
I20260810 01:46:49.311689 12139 catalog_manager.cc:4841] TS 40a6de2a6f85463998b46a214b210bb2 (127.9.77.129:39511): tablet 516a6ba7a82a48ceb598840bd2a0464e (table test-table [id=3a8132c84b05467e93ebf061da872034]) successfully deleted
I20260810 01:46:49.311740 12582 tablet_service.cc:1514] Processing DeleteTablet for tablet 516a6ba7a82a48ceb598840bd2a0464e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:46:49 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:33368
I20260810 01:46:49.312070 13043 tablet_replica.cc:330] T 516a6ba7a82a48ceb598840bd2a0464e P 9e29676918e746de84c4e8b62f658f35: stopping tablet replica
I20260810 01:46:49.312140 12141 catalog_manager.cc:4841] TS e7a50c0229de40a6bc28bd32ea6649ef (127.9.77.131:35103): tablet 516a6ba7a82a48ceb598840bd2a0464e (table test-table [id=3a8132c84b05467e93ebf061da872034]) successfully deleted
I20260810 01:46:49.312225 13043 raft_consensus.cc:2229] T 516a6ba7a82a48ceb598840bd2a0464e P 9e29676918e746de84c4e8b62f658f35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:46:49.312382 13043 raft_consensus.cc:2258] T 516a6ba7a82a48ceb598840bd2a0464e P 9e29676918e746de84c4e8b62f658f35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:46:49.313465 13043 ts_tablet_manager.cc:1905] T 516a6ba7a82a48ceb598840bd2a0464e P 9e29676918e746de84c4e8b62f658f35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:46:49.316396 12375 tablet_service.cc:1467] Processing CreateTablet for tablet ccb6652014154705a9601dc69bdb140f (DEFAULT_TABLE table=test-table [id=5944dfb269594772804bfeb720b2c40c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:46:49.316634 12786 tablet_service.cc:1467] Processing CreateTablet for tablet ccb6652014154705a9601dc69bdb140f (DEFAULT_TABLE table=test-table [id=5944dfb269594772804bfeb720b2c40c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:46:49.316785 12375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb6652014154705a9601dc69bdb140f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:49.316962 12786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb6652014154705a9601dc69bdb140f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:49.317327 13043 ts_tablet_manager.cc:1918] T 516a6ba7a82a48ceb598840bd2a0464e P 9e29676918e746de84c4e8b62f658f35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 01:46:49.317413 13043 log.cc:1196] T 516a6ba7a82a48ceb598840bd2a0464e P 9e29676918e746de84c4e8b62f658f35: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-1/wal/wals/516a6ba7a82a48ceb598840bd2a0464e
I20260810 01:46:49.317711 13043 ts_tablet_manager.cc:1939] T 516a6ba7a82a48ceb598840bd2a0464e P 9e29676918e746de84c4e8b62f658f35: Deleting consensus metadata
I20260810 01:46:49.319816 13062 tablet_bootstrap.cc:492] T ccb6652014154705a9601dc69bdb140f P e7a50c0229de40a6bc28bd32ea6649ef: Bootstrap starting.
I20260810 01:46:49.320484 12143 catalog_manager.cc:4841] TS 9e29676918e746de84c4e8b62f658f35 (127.9.77.130:45423): tablet 516a6ba7a82a48ceb598840bd2a0464e (table test-table [id=3a8132c84b05467e93ebf061da872034]) successfully deleted
I20260810 01:46:49.321046 13062 tablet_bootstrap.cc:654] T ccb6652014154705a9601dc69bdb140f P e7a50c0229de40a6bc28bd32ea6649ef: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:49.321234 12582 tablet_service.cc:1467] Processing CreateTablet for tablet ccb6652014154705a9601dc69bdb140f (DEFAULT_TABLE table=test-table [id=5944dfb269594772804bfeb720b2c40c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:46:49.321547 12582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccb6652014154705a9601dc69bdb140f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:49.321789 13060 tablet_bootstrap.cc:492] T ccb6652014154705a9601dc69bdb140f P 40a6de2a6f85463998b46a214b210bb2: Bootstrap starting.
I20260810 01:46:49.322942 13060 tablet_bootstrap.cc:654] T ccb6652014154705a9601dc69bdb140f P 40a6de2a6f85463998b46a214b210bb2: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:49.324481 13062 tablet_bootstrap.cc:492] T ccb6652014154705a9601dc69bdb140f P e7a50c0229de40a6bc28bd32ea6649ef: No bootstrap required, opened a new log
I20260810 01:46:49.324617 13062 ts_tablet_manager.cc:1397] T ccb6652014154705a9601dc69bdb140f P e7a50c0229de40a6bc28bd32ea6649ef: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260810 01:46:49.325438 13062 raft_consensus.cc:348] T ccb6652014154705a9601dc69bdb140f P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:49.325610 13062 raft_consensus.cc:374] T ccb6652014154705a9601dc69bdb140f P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:49.325657 13062 raft_consensus.cc:729] T ccb6652014154705a9601dc69bdb140f P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a50c0229de40a6bc28bd32ea6649ef, State: Initialized, Role: FOLLOWER
I20260810 01:46:49.326377 13059 tablet_bootstrap.cc:492] T ccb6652014154705a9601dc69bdb140f P 9e29676918e746de84c4e8b62f658f35: Bootstrap starting.
I20260810 01:46:49.326433 13060 tablet_bootstrap.cc:492] T ccb6652014154705a9601dc69bdb140f P 40a6de2a6f85463998b46a214b210bb2: No bootstrap required, opened a new log
I20260810 01:46:49.326547 13060 ts_tablet_manager.cc:1397] T ccb6652014154705a9601dc69bdb140f P 40a6de2a6f85463998b46a214b210bb2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260810 01:46:49.326990 13062 consensus_queue.cc:260] T ccb6652014154705a9601dc69bdb140f P e7a50c0229de40a6bc28bd32ea6649ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:49.327121 13060 raft_consensus.cc:348] T ccb6652014154705a9601dc69bdb140f P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:49.327273 13060 raft_consensus.cc:374] T ccb6652014154705a9601dc69bdb140f P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:49.327309 13060 raft_consensus.cc:729] T ccb6652014154705a9601dc69bdb140f P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40a6de2a6f85463998b46a214b210bb2, State: Initialized, Role: FOLLOWER
I20260810 01:46:49.327394 13059 tablet_bootstrap.cc:654] T ccb6652014154705a9601dc69bdb140f P 9e29676918e746de84c4e8b62f658f35: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:49.327509 13060 consensus_queue.cc:260] T ccb6652014154705a9601dc69bdb140f P 40a6de2a6f85463998b46a214b210bb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:49.327956 13060 ts_tablet_manager.cc:1428] T ccb6652014154705a9601dc69bdb140f P 40a6de2a6f85463998b46a214b210bb2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 01:46:49.328917 13062 ts_tablet_manager.cc:1428] T ccb6652014154705a9601dc69bdb140f P e7a50c0229de40a6bc28bd32ea6649ef: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260810 01:46:49.330643 13059 tablet_bootstrap.cc:492] T ccb6652014154705a9601dc69bdb140f P 9e29676918e746de84c4e8b62f658f35: No bootstrap required, opened a new log
I20260810 01:46:49.330765 13059 ts_tablet_manager.cc:1397] T ccb6652014154705a9601dc69bdb140f P 9e29676918e746de84c4e8b62f658f35: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 01:46:49.331274 13059 raft_consensus.cc:348] T ccb6652014154705a9601dc69bdb140f P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:49.331434 13059 raft_consensus.cc:374] T ccb6652014154705a9601dc69bdb140f P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:49.331472 13059 raft_consensus.cc:729] T ccb6652014154705a9601dc69bdb140f P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e29676918e746de84c4e8b62f658f35, State: Initialized, Role: FOLLOWER
I20260810 01:46:49.331594 13059 consensus_queue.cc:260] T ccb6652014154705a9601dc69bdb140f P 9e29676918e746de84c4e8b62f658f35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:49.331913 13059 ts_tablet_manager.cc:1428] T ccb6652014154705a9601dc69bdb140f P 9e29676918e746de84c4e8b62f658f35: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 01:46:49.496438 13069 raft_consensus.cc:482] T ccb6652014154705a9601dc69bdb140f P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:46:49.496658 13069 raft_consensus.cc:504] T ccb6652014154705a9601dc69bdb140f P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:49.497192 13069 leader_election.cc:288] T ccb6652014154705a9601dc69bdb140f P 40a6de2a6f85463998b46a214b210bb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e29676918e746de84c4e8b62f658f35 (127.9.77.130:45423), e7a50c0229de40a6bc28bd32ea6649ef (127.9.77.131:35103)
I20260810 01:46:49.497965 12819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb6652014154705a9601dc69bdb140f" candidate_uuid: "40a6de2a6f85463998b46a214b210bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" is_pre_election: true
I20260810 01:46:49.497967 12611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb6652014154705a9601dc69bdb140f" candidate_uuid: "40a6de2a6f85463998b46a214b210bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e29676918e746de84c4e8b62f658f35" is_pre_election: true
I20260810 01:46:49.498214 12819 raft_consensus.cc:2454] T ccb6652014154705a9601dc69bdb140f P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40a6de2a6f85463998b46a214b210bb2 in term 0.
I20260810 01:46:49.498214 12611 raft_consensus.cc:2454] T ccb6652014154705a9601dc69bdb140f P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40a6de2a6f85463998b46a214b210bb2 in term 0.
I20260810 01:46:49.498660 12301 leader_election.cc:302] T ccb6652014154705a9601dc69bdb140f P 40a6de2a6f85463998b46a214b210bb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40a6de2a6f85463998b46a214b210bb2, e7a50c0229de40a6bc28bd32ea6649ef; no voters:
I20260810 01:46:49.498919 13069 raft_consensus.cc:2781] T ccb6652014154705a9601dc69bdb140f P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:46:49.498997 13069 raft_consensus.cc:482] T ccb6652014154705a9601dc69bdb140f P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:46:49.499032 13069 raft_consensus.cc:3037] T ccb6652014154705a9601dc69bdb140f P 40a6de2a6f85463998b46a214b210bb2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:49.500550 13069 raft_consensus.cc:504] T ccb6652014154705a9601dc69bdb140f P 40a6de2a6f85463998b46a214b210bb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:49.501041 13069 leader_election.cc:288] T ccb6652014154705a9601dc69bdb140f P 40a6de2a6f85463998b46a214b210bb2 [CANDIDATE]: Term 1 election: Requested vote from peers 9e29676918e746de84c4e8b62f658f35 (127.9.77.130:45423), e7a50c0229de40a6bc28bd32ea6649ef (127.9.77.131:35103)
I20260810 01:46:49.501600 12819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb6652014154705a9601dc69bdb140f" candidate_uuid: "40a6de2a6f85463998b46a214b210bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a50c0229de40a6bc28bd32ea6649ef"
I20260810 01:46:49.501600 12611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccb6652014154705a9601dc69bdb140f" candidate_uuid: "40a6de2a6f85463998b46a214b210bb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e29676918e746de84c4e8b62f658f35"
I20260810 01:46:49.501770 12819 raft_consensus.cc:3037] T ccb6652014154705a9601dc69bdb140f P e7a50c0229de40a6bc28bd32ea6649ef [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:49.501780 12611 raft_consensus.cc:3037] T ccb6652014154705a9601dc69bdb140f P 9e29676918e746de84c4e8b62f658f35 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:49.503206 12611 raft_consensus.cc:2454] T ccb6652014154705a9601dc69bdb140f P 9e29676918e746de84c4e8b62f658f35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40a6de2a6f85463998b46a214b210bb2 in term 1.
I20260810 01:46:49.503206 12819 raft_consensus.cc:2454] T ccb6652014154705a9601dc69bdb140f P e7a50c0229de40a6bc28bd32ea6649ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40a6de2a6f85463998b46a214b210bb2 in term 1.
I20260810 01:46:49.503566 12301 leader_election.cc:302] T ccb6652014154705a9601dc69bdb140f P 40a6de2a6f85463998b46a214b210bb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40a6de2a6f85463998b46a214b210bb2, e7a50c0229de40a6bc28bd32ea6649ef; no voters:
I20260810 01:46:49.503842 13069 raft_consensus.cc:2781] T ccb6652014154705a9601dc69bdb140f P 40a6de2a6f85463998b46a214b210bb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:46:49.503978 13069 raft_consensus.cc:686] T ccb6652014154705a9601dc69bdb140f P 40a6de2a6f85463998b46a214b210bb2 [term 1 LEADER]: Becoming Leader. State: Replica: 40a6de2a6f85463998b46a214b210bb2, State: Running, Role: LEADER
I20260810 01:46:49.504192 13069 consensus_queue.cc:237] T ccb6652014154705a9601dc69bdb140f P 40a6de2a6f85463998b46a214b210bb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } }
I20260810 01:46:49.506719 12163 catalog_manager.cc:5495] T ccb6652014154705a9601dc69bdb140f P 40a6de2a6f85463998b46a214b210bb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40a6de2a6f85463998b46a214b210bb2 (127.9.77.129). New cstate: current_term: 1 leader_uuid: "40a6de2a6f85463998b46a214b210bb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40a6de2a6f85463998b46a214b210bb2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 } health_report { overall_health: UNKNOWN } } }
I20260810 01:46:49.540285 12819 raft_consensus.cc:1261] T ccb6652014154705a9601dc69bdb140f P e7a50c0229de40a6bc28bd32ea6649ef [term 1 FOLLOWER]: Refusing update from remote peer 40a6de2a6f85463998b46a214b210bb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:46:49.540295 12611 raft_consensus.cc:1261] T ccb6652014154705a9601dc69bdb140f P 9e29676918e746de84c4e8b62f658f35 [term 1 FOLLOWER]: Refusing update from remote peer 40a6de2a6f85463998b46a214b210bb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:46:49.541025 13083 consensus_queue.cc:1035] T ccb6652014154705a9601dc69bdb140f P 40a6de2a6f85463998b46a214b210bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7a50c0229de40a6bc28bd32ea6649ef" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:46:49.541322 13069 consensus_queue.cc:1035] T ccb6652014154705a9601dc69bdb140f P 40a6de2a6f85463998b46a214b210bb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e29676918e746de84c4e8b62f658f35" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 01:46:49.558745 13020 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 01:46:51.120980 12163 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:46:51.121199 12163 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:46:51.124187 12163 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad82f728b5a5494a9f9deaf9c241c54f: Sending DeleteTablet for 3 replicas of tablet ccb6652014154705a9601dc69bdb140f
I20260810 01:46:51.124804 12582 tablet_service.cc:1514] Processing DeleteTablet for tablet ccb6652014154705a9601dc69bdb140f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:46:51 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:33368
I20260810 01:46:51.124822 12375 tablet_service.cc:1514] Processing DeleteTablet for tablet ccb6652014154705a9601dc69bdb140f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:46:51 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:35166
I20260810 01:46:51.124955 12786 tablet_service.cc:1514] Processing DeleteTablet for tablet ccb6652014154705a9601dc69bdb140f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:46:51 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:54418
I20260810 01:46:51.125324 13139 tablet_replica.cc:330] T ccb6652014154705a9601dc69bdb140f P 40a6de2a6f85463998b46a214b210bb2: stopping tablet replica
I20260810 01:46:51.125384 13141 tablet_replica.cc:330] T ccb6652014154705a9601dc69bdb140f P e7a50c0229de40a6bc28bd32ea6649ef: stopping tablet replica
I20260810 01:46:51.125370 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 12275
I20260810 01:46:51.134469 13141 raft_consensus.cc:2229] T ccb6652014154705a9601dc69bdb140f P e7a50c0229de40a6bc28bd32ea6649ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:46:51.134550 13138 tablet_replica.cc:330] T ccb6652014154705a9601dc69bdb140f P 9e29676918e746de84c4e8b62f658f35: stopping tablet replica
I20260810 01:46:51.134758 13138 raft_consensus.cc:2229] T ccb6652014154705a9601dc69bdb140f P 9e29676918e746de84c4e8b62f658f35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:46:51.134922 13141 raft_consensus.cc:2258] T ccb6652014154705a9601dc69bdb140f P e7a50c0229de40a6bc28bd32ea6649ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:46:51.135031 13138 raft_consensus.cc:2258] T ccb6652014154705a9601dc69bdb140f P 9e29676918e746de84c4e8b62f658f35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:46:51.136761 13138 ts_tablet_manager.cc:1905] T ccb6652014154705a9601dc69bdb140f P 9e29676918e746de84c4e8b62f658f35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:46:51.136806 13141 ts_tablet_manager.cc:1905] T ccb6652014154705a9601dc69bdb140f P e7a50c0229de40a6bc28bd32ea6649ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:46:51.139675 13138 ts_tablet_manager.cc:1918] T ccb6652014154705a9601dc69bdb140f P 9e29676918e746de84c4e8b62f658f35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 01:46:51.139786 13138 log.cc:1196] T ccb6652014154705a9601dc69bdb140f P 9e29676918e746de84c4e8b62f658f35: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-1/wal/wals/ccb6652014154705a9601dc69bdb140f
I20260810 01:46:51.140156 13138 ts_tablet_manager.cc:1939] T ccb6652014154705a9601dc69bdb140f P 9e29676918e746de84c4e8b62f658f35: Deleting consensus metadata
I20260810 01:46:51.140658 13141 ts_tablet_manager.cc:1918] T ccb6652014154705a9601dc69bdb140f P e7a50c0229de40a6bc28bd32ea6649ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 01:46:51.140755 13141 log.cc:1196] T ccb6652014154705a9601dc69bdb140f P e7a50c0229de40a6bc28bd32ea6649ef: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786326392877349-9526-0/minicluster-data/ts-2/wal/wals/ccb6652014154705a9601dc69bdb140f
I20260810 01:46:51.141047 13141 ts_tablet_manager.cc:1939] T ccb6652014154705a9601dc69bdb140f P e7a50c0229de40a6bc28bd32ea6649ef: Deleting consensus metadata
I20260810 01:46:51.141185 12143 catalog_manager.cc:4841] TS 9e29676918e746de84c4e8b62f658f35 (127.9.77.130:45423): tablet ccb6652014154705a9601dc69bdb140f (table test-table [id=5944dfb269594772804bfeb720b2c40c]) successfully deleted
I20260810 01:46:51.141986 12141 catalog_manager.cc:4841] TS e7a50c0229de40a6bc28bd32ea6649ef (127.9.77.131:35103): tablet ccb6652014154705a9601dc69bdb140f (table test-table [id=5944dfb269594772804bfeb720b2c40c]) successfully deleted
W20260810 01:46:51.147348 12139 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.77.129:39511 (error 108)
I20260810 01:46:51.147714 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 12483
I20260810 01:46:51.156970 13143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:51.155748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:39511 (local address 127.0.0.1:35194)
0810 01:46:51.156720 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:51.156818 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.129:39511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":873,"thread_start_us":618,"threads_started":1}
W20260810 01:46:51.157166 12139 catalog_manager.cc:4584] TS 40a6de2a6f85463998b46a214b210bb2 (127.9.77.129:39511): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ccb6652014154705a9601dc69bdb140f: Network error: Client connection negotiation failed: client connection to 127.9.77.129:39511: connect: Connection refused (error 111)
I20260810 01:46:51.165023 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 12700
I20260810 01:46:51.182065 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 12112
2026-08-10T01:46:51Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7124 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:46:51 dist-test-slave-kvfs krb5kdc[13147](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:46:51 dist-test-slave-kvfs krb5kdc[13147](info): set up 2 sockets
Aug 10 01:46:51 dist-test-slave-kvfs krb5kdc[13147](info): commencing operation
krb5kdc: starting...
W20260810 01:46:52.516682 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.296s user 0.006s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:46:52 dist-test-slave-kvfs krb5kdc[13147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:46:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:46:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:46:52.660341 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:38443
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:38407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:38443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:46:52.794080 13165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:46:52.794435 13165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:46:52.794480 13165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:46:52.798681 13165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:46:52.798745 13165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:46:52.798764 13165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:46:52.798781 13165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:46:52.798796 13165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:46:52.804466 13165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:38407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:38443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:38443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:46:52.805948 13165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:46:52.807260 13165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:46:52.817903 13174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:52.817885 13176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:52.817854 13173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:52.817857 13165 server_base.cc:1034] running on GCE node
I20260810 01:46:52.819005 13165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:46:52.820559 13165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:46:52.821760 13165 hybrid_clock.cc:648] HybridClock initialized: now 1786326412821750 us; error 33 us; skew 500 ppm
Aug 10 01:46:52 dist-test-slave-kvfs krb5kdc[13147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326412, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:46:52.825074 13165 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:46:52.826220 13165 webserver.cc:460] Webserver started at http://127.9.77.190:36455/ using document root <none> and password file <none>
I20260810 01:46:52.826934 13165 fs_manager.cc:359] Metadata directory not provided
I20260810 01:46:52.827033 13165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:46:52.827319 13165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:46:52.829916 13165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "96acb0d15d9e4c9795a83ffb5ff76475"
format_stamp: "Formatted at 2026-08-10 01:46:52 on dist-test-slave-kvfs"
server_key: "3bfa936556ec45c8b94f68dcd8a6178f"
server_key_iv: "de4bf6943dfd6551ef8a807b0cd4c69e"
server_key_version: "encryptionkey@0"
I20260810 01:46:52.830592 13165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "96acb0d15d9e4c9795a83ffb5ff76475"
format_stamp: "Formatted at 2026-08-10 01:46:52 on dist-test-slave-kvfs"
server_key: "3bfa936556ec45c8b94f68dcd8a6178f"
server_key_iv: "de4bf6943dfd6551ef8a807b0cd4c69e"
server_key_version: "encryptionkey@0"
I20260810 01:46:52.835135 13165 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 01:46:52.838039 13187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:52.839542 13165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 01:46:52.839704 13165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "96acb0d15d9e4c9795a83ffb5ff76475"
format_stamp: "Formatted at 2026-08-10 01:46:52 on dist-test-slave-kvfs"
server_key: "3bfa936556ec45c8b94f68dcd8a6178f"
server_key_iv: "de4bf6943dfd6551ef8a807b0cd4c69e"
server_key_version: "encryptionkey@0"
I20260810 01:46:52.839835 13165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:46:52.855852 13165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:46:52.859534 13165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:46:52.859793 13165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:46:52.868633 13165 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:38443
I20260810 01:46:52.868662 13273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:38443 every 8 connection(s)
I20260810 01:46:52.869908 13165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:46:52.873765 13274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:52.879904 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 13165
I20260810 01:46:52.880112 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:46:52.880445 9526 external_mini_cluster.cc:1428] Setting key 11d0b94f7cc66fe2936542f6f28c3da5
I20260810 01:46:52.881500 13274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96acb0d15d9e4c9795a83ffb5ff76475: Bootstrap starting.
I20260810 01:46:52.884406 13274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96acb0d15d9e4c9795a83ffb5ff76475: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:52.885560 13274 log.cc:824] T 00000000000000000000000000000000 P 96acb0d15d9e4c9795a83ffb5ff76475: Log is configured to *not* fsync() on all Append() calls
I20260810 01:46:52.888449 13274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96acb0d15d9e4c9795a83ffb5ff76475: No bootstrap required, opened a new log
Aug 10 01:46:52 dist-test-slave-kvfs krb5kdc[13147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:52.893747 13274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96acb0d15d9e4c9795a83ffb5ff76475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96acb0d15d9e4c9795a83ffb5ff76475" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 38443 } }
I20260810 01:46:52.894033 13274 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96acb0d15d9e4c9795a83ffb5ff76475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:52.894106 13274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96acb0d15d9e4c9795a83ffb5ff76475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96acb0d15d9e4c9795a83ffb5ff76475, State: Initialized, Role: FOLLOWER
I20260810 01:46:52.894693 13274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96acb0d15d9e4c9795a83ffb5ff76475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96acb0d15d9e4c9795a83ffb5ff76475" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 38443 } }
I20260810 01:46:52.894884 13274 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96acb0d15d9e4c9795a83ffb5ff76475 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:46:52.894970 13274 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96acb0d15d9e4c9795a83ffb5ff76475 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:46:52.895107 13274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96acb0d15d9e4c9795a83ffb5ff76475 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:52.896486 13274 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96acb0d15d9e4c9795a83ffb5ff76475 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96acb0d15d9e4c9795a83ffb5ff76475" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 38443 } }
I20260810 01:46:52.896972 13274 leader_election.cc:302] T 00000000000000000000000000000000 P 96acb0d15d9e4c9795a83ffb5ff76475 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96acb0d15d9e4c9795a83ffb5ff76475; no voters:
I20260810 01:46:52.897414 13274 leader_election.cc:288] T 00000000000000000000000000000000 P 96acb0d15d9e4c9795a83ffb5ff76475 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:46:52.897552 13279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96acb0d15d9e4c9795a83ffb5ff76475 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:46:52.897827 13279 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96acb0d15d9e4c9795a83ffb5ff76475 [term 1 LEADER]: Becoming Leader. State: Replica: 96acb0d15d9e4c9795a83ffb5ff76475, State: Running, Role: LEADER
I20260810 01:46:52.898255 13279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96acb0d15d9e4c9795a83ffb5ff76475 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96acb0d15d9e4c9795a83ffb5ff76475" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 38443 } }
I20260810 01:46:52.898602 13277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:52.882287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38722 (local address 127.9.77.190:38443)
0810 01:46:52.882819 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:52.882838 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:52.882881 (+ 43us) server_negotiation.cc:407] Connection header received
0810 01:46:52.883601 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:52.883622 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:52.884083 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:52.884434 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:52.885539 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:52.887071 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:52.888403 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:52.889028 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:52.889167 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:52.892423 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:52.892430 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:52.892448 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:52.892480 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:52.894669 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:52.895295 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:52.895303 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:52.895313 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:52.895480 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:52.895809 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:52.895812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:52.895814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:52.896479 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:52.896724 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:52.897006 (+ 282us) server_negotiation.cc:299] Negotiation successful
0810 01:46:52.897409 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":172,"threads_started":1}
I20260810 01:46:52.898980 13274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96acb0d15d9e4c9795a83ffb5ff76475 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:46:52.900584 13282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96acb0d15d9e4c9795a83ffb5ff76475 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96acb0d15d9e4c9795a83ffb5ff76475. Latest consensus state: current_term: 1 leader_uuid: "96acb0d15d9e4c9795a83ffb5ff76475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96acb0d15d9e4c9795a83ffb5ff76475" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 38443 } } }
I20260810 01:46:52.900732 13282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96acb0d15d9e4c9795a83ffb5ff76475 [sys.catalog]: This master's current role is: LEADER
I20260810 01:46:52.901052 13280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96acb0d15d9e4c9795a83ffb5ff76475 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96acb0d15d9e4c9795a83ffb5ff76475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96acb0d15d9e4c9795a83ffb5ff76475" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 38443 } } }
I20260810 01:46:52.901166 13280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96acb0d15d9e4c9795a83ffb5ff76475 [sys.catalog]: This master's current role is: LEADER
W20260810 01:46:52.905438 13301 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 96acb0d15d9e4c9795a83ffb5ff76475: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 01:46:52.905512 13301 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 01:46:52.905597 13294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:46:52.906337 13294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:46:52.912874 13294 catalog_manager.cc:1347] Generated new cluster ID: 1362e081377948038a2f17245a934d91
I20260810 01:46:52.912954 13294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:46:52.932749 13294 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:46:52.934365 13294 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:46:52.949343 13294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96acb0d15d9e4c9795a83ffb5ff76475: Generated new TSK 0
I20260810 01:46:52.950702 13294 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.77.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
I20260810 01:46:52.987341 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:0
--local_ip_for_outbound_sockets=127.9.77.129
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:38443
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:38407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:46:53.126269 13319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:46:53.126610 13319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:46:53.126658 13319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:46:53.130795 13319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:46:53.130869 13319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:46:53.130935 13319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:46:53.136579 13319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:38407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:38443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:46:53.138000 13319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:46:53.139319 13319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:46:53.151386 13330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:53.151363 13326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:53.151466 13319 server_base.cc:1034] running on GCE node
W20260810 01:46:53.151368 13325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:53.152410 13319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:46:53.153429 13319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:46:53.154661 13319 hybrid_clock.cc:648] HybridClock initialized: now 1786326413154643 us; error 28 us; skew 500 ppm
Aug 10 01:46:53 dist-test-slave-kvfs krb5kdc[13147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326413, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:46:53.158277 13319 init.cc:377] Logged in from keytab as kudu/127.9.77.129@KRBTEST.COM (short username kudu)
I20260810 01:46:53.159411 13319 webserver.cc:460] Webserver started at http://127.9.77.129:35323/ using document root <none> and password file <none>
I20260810 01:46:53.160223 13319 fs_manager.cc:359] Metadata directory not provided
I20260810 01:46:53.160331 13319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:46:53.160624 13319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:46:53.163259 13319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/data/instance:
uuid: "30d0135c3f0c43a3ae3bab9ef9749da3"
format_stamp: "Formatted at 2026-08-10 01:46:53 on dist-test-slave-kvfs"
server_key: "e18ef2d7585829a4a1fec52a4beac3d4"
server_key_iv: "db72d797153346fcc3934f681e7fc17d"
server_key_version: "encryptionkey@0"
I20260810 01:46:53.163988 13319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance:
uuid: "30d0135c3f0c43a3ae3bab9ef9749da3"
format_stamp: "Formatted at 2026-08-10 01:46:53 on dist-test-slave-kvfs"
server_key: "e18ef2d7585829a4a1fec52a4beac3d4"
server_key_iv: "db72d797153346fcc3934f681e7fc17d"
server_key_version: "encryptionkey@0"
I20260810 01:46:53.168816 13319 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 01:46:53.172197 13339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:53.173683 13319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:46:53.173823 13319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "30d0135c3f0c43a3ae3bab9ef9749da3"
format_stamp: "Formatted at 2026-08-10 01:46:53 on dist-test-slave-kvfs"
server_key: "e18ef2d7585829a4a1fec52a4beac3d4"
server_key_iv: "db72d797153346fcc3934f681e7fc17d"
server_key_version: "encryptionkey@0"
I20260810 01:46:53.173933 13319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:46:53.186091 13319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:46:53.189826 13319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:46:53.189998 13319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:46:53.190623 13319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:46:53.191540 13319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:46:53.191589 13319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:53.191633 13319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:46:53.191654 13319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:53.202664 13319 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:41655
I20260810 01:46:53.202682 13505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:41655 every 8 connection(s)
I20260810 01:46:53.203902 13319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:46:53.206568 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 13319
I20260810 01:46:53.206717 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance
I20260810 01:46:53.207110 9526 external_mini_cluster.cc:1428] Setting key cba4d8fd7272038e8bd4ef0061c0e9fe
Aug 10 01:46:53 dist-test-slave-kvfs krb5kdc[13147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326413, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:53.221776 13277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:53.206333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:34519 (local address 127.9.77.190:38443)
0810 01:46:53.206513 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:53.206517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:53.207754 (+ 1237us) server_negotiation.cc:407] Connection header received
0810 01:46:53.209688 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:53.209692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:53.209754 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:53.209866 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:53.211301 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:53.211892 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:53.213244 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:53.213506 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:53.213618 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:53.217046 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:53.217049 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:53.217051 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:53.217079 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:53.218991 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:53.219579 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:53.219583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:53.219585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:53.219695 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:53.220161 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:53.220167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:53.220170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:53.220496 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:53.220614 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:53.221346 (+ 732us) server_negotiation.cc:299] Negotiation successful
0810 01:46:53.221555 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
WARNING: no policy specified for kudu/127.9.77.130@KRBTEST.COM; defaulting to no policy
I20260810 01:46:53.222800 13511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:53.206906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:38443 (local address 127.9.77.129:34519)
0810 01:46:53.207527 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:53.207579 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:53.209445 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:53.210088 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:53.210096 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:53.210532 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:46:53.211132 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:53.211147 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:53.212095 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:53.212099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:53.213102 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:53.213111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:53.213681 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:53.213686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:53.213970 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:53.214739 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:46:53.214767 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:46:53.216837 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:46:53.219127 (+ 2290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:53.219133 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:53.219150 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:53.219434 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:53.219814 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:53.219817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:53.219819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:53.219980 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:53.220647 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:46:53.220654 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:46:53.220995 (+ 341us) client_negotiation.cc:769] Sending connection context
0810 01:46:53.221200 (+ 205us) client_negotiation.cc:240] Negotiation successful
0810 01:46:53.221630 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":155,"threads_started":1}
I20260810 01:46:53.224532 13507 heartbeater.cc:347] Connected to a master server at 127.9.77.190:38443
I20260810 01:46:53.224888 13507 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.130@KRBTEST.COM" created.
I20260810 01:46:53.225629 13507 heartbeater.cc:511] Master 127.9.77.190:38443 requested a full tablet report, sending...
I20260810 01:46:53.227617 13213 ts_manager.cc:194] Registered new tserver with Master: 30d0135c3f0c43a3ae3bab9ef9749da3 (127.9.77.129:41655)
I20260810 01:46:53.229054 13213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.129@KRBTEST.COM'} at 127.9.77.129:34519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
I20260810 01:46:53.244602 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:0
--local_ip_for_outbound_sockets=127.9.77.130
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:38443
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:38407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:46:53.389204 13514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:46:53.389621 13514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:46:53.389706 13514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:46:53.394119 13514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:46:53.394236 13514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:46:53.394345 13514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:46:53.400806 13514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:38407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:38443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:46:53.402531 13514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:46:53.403903 13514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:46:53.416318 13522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:53.416375 13514 server_base.cc:1034] running on GCE node
W20260810 01:46:53.416323 13523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:53.416395 13525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:53.417140 13514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:46:53.417938 13514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:46:53.419133 13514 hybrid_clock.cc:648] HybridClock initialized: now 1786326413419113 us; error 34 us; skew 500 ppm
Aug 10 01:46:53 dist-test-slave-kvfs krb5kdc[13147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326413, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:46:53.422358 13514 init.cc:377] Logged in from keytab as kudu/127.9.77.130@KRBTEST.COM (short username kudu)
I20260810 01:46:53.423456 13514 webserver.cc:460] Webserver started at http://127.9.77.130:42043/ using document root <none> and password file <none>
I20260810 01:46:53.424213 13514 fs_manager.cc:359] Metadata directory not provided
I20260810 01:46:53.424325 13514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:46:53.424604 13514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:46:53.427175 13514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/data/instance:
uuid: "a46055a64e8b48d38b3163dae3667f83"
format_stamp: "Formatted at 2026-08-10 01:46:53 on dist-test-slave-kvfs"
server_key: "cfb2a6673809d50f4e957ea566c5e246"
server_key_iv: "41ae132a76b50d07dbed0018c1fcf78d"
server_key_version: "encryptionkey@0"
I20260810 01:46:53.427860 13514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance:
uuid: "a46055a64e8b48d38b3163dae3667f83"
format_stamp: "Formatted at 2026-08-10 01:46:53 on dist-test-slave-kvfs"
server_key: "cfb2a6673809d50f4e957ea566c5e246"
server_key_iv: "41ae132a76b50d07dbed0018c1fcf78d"
server_key_version: "encryptionkey@0"
I20260810 01:46:53.432296 13514 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 01:46:53.435182 13539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:53.436755 13514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:46:53.436909 13514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "a46055a64e8b48d38b3163dae3667f83"
format_stamp: "Formatted at 2026-08-10 01:46:53 on dist-test-slave-kvfs"
server_key: "cfb2a6673809d50f4e957ea566c5e246"
server_key_iv: "41ae132a76b50d07dbed0018c1fcf78d"
server_key_version: "encryptionkey@0"
I20260810 01:46:53.437053 13514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:46:53.457592 13514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:46:53.461330 13514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:46:53.461514 13514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:46:53.462196 13514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:46:53.463143 13514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:46:53.463192 13514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:53.463238 13514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:46:53.463259 13514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:53.474156 13514 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:35715
I20260810 01:46:53.474180 13704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:35715 every 8 connection(s)
I20260810 01:46:53.475291 13514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:46:53.478051 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 13514
I20260810 01:46:53.478185 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance
I20260810 01:46:53.478575 9526 external_mini_cluster.cc:1428] Setting key e5988c4d1223ff2564bf548f4cefc86c
Aug 10 01:46:53 dist-test-slave-kvfs krb5kdc[13147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326413, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.77.131@KRBTEST.COM; defaulting to no policy
I20260810 01:46:53.493541 13277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:53.477660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:52027 (local address 127.9.77.190:38443)
0810 01:46:53.477830 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:53.477836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:53.479102 (+ 1266us) server_negotiation.cc:407] Connection header received
0810 01:46:53.481101 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:53.481104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:53.481156 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:53.481229 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:53.483102 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:53.483680 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:53.485639 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:53.485875 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:53.485966 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:53.489264 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:53.489271 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:53.489276 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:53.489315 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:53.491093 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:53.491638 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:53.491642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:53.491644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:53.491781 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:53.492209 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:53.492214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:53.492216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:53.492508 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:53.492628 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:53.493245 (+ 617us) server_negotiation.cc:299] Negotiation successful
0810 01:46:53.493384 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 01:46:53.494680 13710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:53.478258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:38443 (local address 127.9.77.130:52027)
0810 01:46:53.478879 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:53.478972 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:53.480845 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:53.481460 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:53.481472 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:53.482096 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:46:53.482935 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:53.482959 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:53.483816 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:53.483822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:53.485419 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:53.485436 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:53.486051 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:53.486058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:53.486333 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:53.487076 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:46:53.487103 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:46:53.489075 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:46:53.491210 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:53.491216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:53.491230 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:53.491510 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:53.491885 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:53.491888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:53.491889 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:53.492090 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:53.492631 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:46:53.492637 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:46:53.492914 (+ 277us) client_negotiation.cc:769] Sending connection context
0810 01:46:53.493138 (+ 224us) client_negotiation.cc:240] Negotiation successful
0810 01:46:53.493454 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":147,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.131@KRBTEST.COM" created.
I20260810 01:46:53.496160 13706 heartbeater.cc:347] Connected to a master server at 127.9.77.190:38443
I20260810 01:46:53.496479 13706 heartbeater.cc:464] Registering TS with master...
I20260810 01:46:53.497093 13706 heartbeater.cc:511] Master 127.9.77.190:38443 requested a full tablet report, sending...
I20260810 01:46:53.498353 13213 ts_manager.cc:194] Registered new tserver with Master: a46055a64e8b48d38b3163dae3667f83 (127.9.77.130:35715)
I20260810 01:46:53.499132 13213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.130@KRBTEST.COM'} at 127.9.77.130:52027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
I20260810 01:46:53.515267 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:0
--local_ip_for_outbound_sockets=127.9.77.131
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:38443
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:38407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:46:53.657244 13716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:46:53.657605 13716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:46:53.657658 13716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:46:53.661906 13716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:46:53.661975 13716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:46:53.662036 13716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:46:53.667829 13716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:38407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:38443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:46:53.669323 13716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:46:53.670614 13716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:46:53.682653 13732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:53.682639 13730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:53.682647 13729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:53.683183 13716 server_base.cc:1034] running on GCE node
I20260810 01:46:53.683626 13716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:46:53.684516 13716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:46:53.685691 13716 hybrid_clock.cc:648] HybridClock initialized: now 1786326413685670 us; error 31 us; skew 500 ppm
Aug 10 01:46:53 dist-test-slave-kvfs krb5kdc[13147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326413, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:46:53.689013 13716 init.cc:377] Logged in from keytab as kudu/127.9.77.131@KRBTEST.COM (short username kudu)
I20260810 01:46:53.690064 13716 webserver.cc:460] Webserver started at http://127.9.77.131:39583/ using document root <none> and password file <none>
I20260810 01:46:53.690773 13716 fs_manager.cc:359] Metadata directory not provided
I20260810 01:46:53.690872 13716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:46:53.691154 13716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:46:53.693768 13716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/data/instance:
uuid: "60bdc24c8c6241a59145a45ef549584a"
format_stamp: "Formatted at 2026-08-10 01:46:53 on dist-test-slave-kvfs"
server_key: "0f519b6f9f7ef538271d231950cec939"
server_key_iv: "040f14cf3472ecb8405ff9692e697ab6"
server_key_version: "encryptionkey@0"
I20260810 01:46:53.694439 13716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance:
uuid: "60bdc24c8c6241a59145a45ef549584a"
format_stamp: "Formatted at 2026-08-10 01:46:53 on dist-test-slave-kvfs"
server_key: "0f519b6f9f7ef538271d231950cec939"
server_key_iv: "040f14cf3472ecb8405ff9692e697ab6"
server_key_version: "encryptionkey@0"
I20260810 01:46:53.699385 13716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260810 01:46:53.702886 13741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:53.704663 13716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 01:46:53.704833 13716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "60bdc24c8c6241a59145a45ef549584a"
format_stamp: "Formatted at 2026-08-10 01:46:53 on dist-test-slave-kvfs"
server_key: "0f519b6f9f7ef538271d231950cec939"
server_key_iv: "040f14cf3472ecb8405ff9692e697ab6"
server_key_version: "encryptionkey@0"
I20260810 01:46:53.704986 13716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:46:53.725667 13716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:46:53.729815 13716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:46:53.730036 13716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:46:53.730823 13716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:46:53.731952 13716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:46:53.732008 13716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:53.732239 13716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:46:53.732265 13716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:53.745453 13716 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:33159
I20260810 01:46:53.745517 13916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:33159 every 8 connection(s)
I20260810 01:46:53.746754 13716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:46:53.755797 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 13716
I20260810 01:46:53.756006 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance
I20260810 01:46:53.756451 9526 external_mini_cluster.cc:1428] Setting key 257bb145b554df120d3709337ae4e313
Aug 10 01:46:53 dist-test-slave-kvfs krb5kdc[13147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326413, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:53.767359 13277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:53.749996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:34413 (local address 127.9.77.190:38443)
0810 01:46:53.750219 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:53.750224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:53.751454 (+ 1230us) server_negotiation.cc:407] Connection header received
0810 01:46:53.753637 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:53.753641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:53.753721 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:53.753816 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:53.755921 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:53.756715 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:53.758371 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:53.758643 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:53.758763 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:53.762553 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:53.762557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:53.762559 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:53.762597 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:53.764380 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:53.765072 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:53.765075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:53.765077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:53.765173 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:53.765648 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:53.765650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:53.765651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:53.765919 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:53.766003 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:53.767004 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0810 01:46:53.767179 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 01:46:53.768726 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:53.750498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:38443 (local address 127.9.77.131:34413)
0810 01:46:53.751195 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:53.751258 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:53.753223 (+ 1965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:53.754327 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:53.754339 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:53.754924 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:46:53.755650 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:53.755670 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:53.756894 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:53.756898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:53.758193 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:53.758207 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:53.758855 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:53.758860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:53.759111 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:53.759988 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:46:53.760010 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:46:53.762275 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:46:53.764556 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:53.764564 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:53.764586 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:53.764959 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:53.765321 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:53.765324 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:53.765326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:53.765545 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:53.766046 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:46:53.766054 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:46:53.766673 (+ 619us) client_negotiation.cc:769] Sending connection context
0810 01:46:53.766913 (+ 240us) client_negotiation.cc:240] Negotiation successful
0810 01:46:53.767324 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":181,"threads_started":1}
I20260810 01:46:53.770450 13919 heartbeater.cc:347] Connected to a master server at 127.9.77.190:38443
I20260810 01:46:53.770835 13919 heartbeater.cc:464] Registering TS with master...
I20260810 01:46:53.771668 13919 heartbeater.cc:511] Master 127.9.77.190:38443 requested a full tablet report, sending...
I20260810 01:46:53.773072 13213 ts_manager.cc:194] Registered new tserver with Master: 60bdc24c8c6241a59145a45ef549584a (127.9.77.131:33159)
I20260810 01:46:53.773869 13213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.131@KRBTEST.COM'} at 127.9.77.131:34413
I20260810 01:46:53.784577 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 01:46:53.806005 13277 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:46:53.804126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38724 (local address 127.9.77.190:38443)
0810 01:46:53.804297 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:53.804301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:53.804505 (+ 204us) server_negotiation.cc:407] Connection header received
0810 01:46:53.804659 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:53.804662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:53.804717 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:53.804807 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:53.805759 (+ 952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38724: BlockingRecv error: recv got EOF from 127.0.0.1:38724 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 01:46:53.808624 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 13319
I20260810 01:46:53.847467 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 13514
I20260810 01:46:53.863883 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 13716
I20260810 01:46:53.879371 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 13165
2026-08-10T01:46:53Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2691 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:46:53 dist-test-slave-kvfs krb5kdc[13936](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:46:53 dist-test-slave-kvfs krb5kdc[13936](info): set up 2 sockets
Aug 10 01:46:53 dist-test-slave-kvfs krb5kdc[13936](info): commencing operation
krb5kdc: starting...
W20260810 01:46:55.166188 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.254s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:46:55 dist-test-slave-kvfs krb5kdc[13936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:46:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:46:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:46:55.316224 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:46815
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:42779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:46815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:46:55.451412 13962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:46:55.451865 13962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:46:55.451953 13962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:46:55.456382 13962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:46:55.456518 13962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:46:55.456557 13962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:46:55.456604 13962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:46:55.456641 13962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:46:55.462607 13962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:42779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:46815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:46815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:46:55.464171 13962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:46:55.465546 13962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:46:55.476219 13970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:55.476245 13972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:55.476344 13962 server_base.cc:1034] running on GCE node
W20260810 01:46:55.476219 13975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:55.477411 13962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:46:55.478791 13962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:46:55.480013 13962 hybrid_clock.cc:648] HybridClock initialized: now 1786326415479963 us; error 63 us; skew 500 ppm
Aug 10 01:46:55 dist-test-slave-kvfs krb5kdc[13936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326415, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:46:55.487574 13962 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:46:55.488842 13962 webserver.cc:460] Webserver started at http://127.9.77.190:35051/ using document root <none> and password file <none>
I20260810 01:46:55.489559 13962 fs_manager.cc:359] Metadata directory not provided
I20260810 01:46:55.489661 13962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:46:55.489959 13962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:46:55.492624 13962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "5e1c2fa7294c44dc812bcb858e4f1196"
format_stamp: "Formatted at 2026-08-10 01:46:55 on dist-test-slave-kvfs"
server_key: "4606b4cc4c3c7097d5f094b012f82d27"
server_key_iv: "173cb66e1134bf30d1116639cd1d6f1f"
server_key_version: "encryptionkey@0"
I20260810 01:46:55.493312 13962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "5e1c2fa7294c44dc812bcb858e4f1196"
format_stamp: "Formatted at 2026-08-10 01:46:55 on dist-test-slave-kvfs"
server_key: "4606b4cc4c3c7097d5f094b012f82d27"
server_key_iv: "173cb66e1134bf30d1116639cd1d6f1f"
server_key_version: "encryptionkey@0"
I20260810 01:46:55.498013 13962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 01:46:55.501091 13985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:55.502620 13962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:46:55.502804 13962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "5e1c2fa7294c44dc812bcb858e4f1196"
format_stamp: "Formatted at 2026-08-10 01:46:55 on dist-test-slave-kvfs"
server_key: "4606b4cc4c3c7097d5f094b012f82d27"
server_key_iv: "173cb66e1134bf30d1116639cd1d6f1f"
server_key_version: "encryptionkey@0"
I20260810 01:46:55.502950 13962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:46:55.516544 13962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:46:55.520594 13962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:46:55.520839 13962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:46:55.530187 13962 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:46815
I20260810 01:46:55.530537 14069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:46815 every 8 connection(s)
I20260810 01:46:55.531519 13962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:46:55.535472 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 13962
I20260810 01:46:55.535624 14070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:55.535636 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:46:55.536057 9526 external_mini_cluster.cc:1428] Setting key 6c2c9ee666165abdffdabe9a38d2070d
I20260810 01:46:55.544605 14070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e1c2fa7294c44dc812bcb858e4f1196: Bootstrap starting.
Aug 10 01:46:55 dist-test-slave-kvfs krb5kdc[13936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:55.548056 14070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e1c2fa7294c44dc812bcb858e4f1196: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:55.549077 14070 log.cc:824] T 00000000000000000000000000000000 P 5e1c2fa7294c44dc812bcb858e4f1196: Log is configured to *not* fsync() on all Append() calls
I20260810 01:46:55.551492 14070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e1c2fa7294c44dc812bcb858e4f1196: No bootstrap required, opened a new log
I20260810 01:46:55.552968 14074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:55.537749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56734 (local address 127.9.77.190:46815)
0810 01:46:55.538248 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:55.538276 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:55.538323 (+ 47us) server_negotiation.cc:407] Connection header received
0810 01:46:55.539107 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:55.539130 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:55.539543 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:55.539896 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:55.541028 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:55.542149 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:55.543426 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:55.543950 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:55.544157 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:55.547504 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:55.547511 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:55.547528 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:55.547564 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:55.549652 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:55.550172 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:55.550178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:55.550183 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:55.550310 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:55.550620 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:55.550624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:55.550626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:55.551105 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:55.551350 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:55.551644 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 01:46:55.551998 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":154,"threads_started":1}
I20260810 01:46:55.555301 14070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e1c2fa7294c44dc812bcb858e4f1196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e1c2fa7294c44dc812bcb858e4f1196" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46815 } }
I20260810 01:46:55.555629 14070 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e1c2fa7294c44dc812bcb858e4f1196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:55.555742 14070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e1c2fa7294c44dc812bcb858e4f1196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1c2fa7294c44dc812bcb858e4f1196, State: Initialized, Role: FOLLOWER
I20260810 01:46:55.556346 14070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e1c2fa7294c44dc812bcb858e4f1196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e1c2fa7294c44dc812bcb858e4f1196" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46815 } }
I20260810 01:46:55.556535 14070 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e1c2fa7294c44dc812bcb858e4f1196 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:46:55.556598 14070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e1c2fa7294c44dc812bcb858e4f1196 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:46:55.556725 14070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e1c2fa7294c44dc812bcb858e4f1196 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:55.556931 9526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 01:46:55.557893 14070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e1c2fa7294c44dc812bcb858e4f1196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e1c2fa7294c44dc812bcb858e4f1196" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46815 } }
I20260810 01:46:55.558292 14070 leader_election.cc:302] T 00000000000000000000000000000000 P 5e1c2fa7294c44dc812bcb858e4f1196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e1c2fa7294c44dc812bcb858e4f1196; no voters:
I20260810 01:46:55.558693 14070 leader_election.cc:288] T 00000000000000000000000000000000 P 5e1c2fa7294c44dc812bcb858e4f1196 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:46:55.558782 14076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e1c2fa7294c44dc812bcb858e4f1196 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:46:55.559017 14076 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e1c2fa7294c44dc812bcb858e4f1196 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1c2fa7294c44dc812bcb858e4f1196, State: Running, Role: LEADER
I20260810 01:46:55.559428 14076 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e1c2fa7294c44dc812bcb858e4f1196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e1c2fa7294c44dc812bcb858e4f1196" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46815 } }
I20260810 01:46:55.560325 14070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e1c2fa7294c44dc812bcb858e4f1196 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:46:55.561707 14077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1c2fa7294c44dc812bcb858e4f1196 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e1c2fa7294c44dc812bcb858e4f1196" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e1c2fa7294c44dc812bcb858e4f1196" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46815 } } }
I20260810 01:46:55.561868 14077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1c2fa7294c44dc812bcb858e4f1196 [sys.catalog]: This master's current role is: LEADER
I20260810 01:46:55.562302 14080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1c2fa7294c44dc812bcb858e4f1196 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e1c2fa7294c44dc812bcb858e4f1196. Latest consensus state: current_term: 1 leader_uuid: "5e1c2fa7294c44dc812bcb858e4f1196" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e1c2fa7294c44dc812bcb858e4f1196" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46815 } } }
I20260810 01:46:55.562407 14080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1c2fa7294c44dc812bcb858e4f1196 [sys.catalog]: This master's current role is: LEADER
I20260810 01:46:55.562404 14092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:46:55.566519 14092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:46:55.574019 14092 catalog_manager.cc:1347] Generated new cluster ID: eae3fec093ff40d08170e08da2b3095a
I20260810 01:46:55.574103 14092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:46:55.584532 14074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:55.578779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56744 (local address 127.9.77.190:46815)
0810 01:46:55.578934 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:55.578940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:55.579229 (+ 289us) server_negotiation.cc:407] Connection header received
0810 01:46:55.579418 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:55.579421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:55.579486 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:55.579581 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:55.580261 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:55.580870 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:55.581782 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:55.582060 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:55.582182 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:55.583331 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:55.583334 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:55.583337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:46:55.583903 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:55.583940 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:46:55.583957 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:55.584012 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:55.584256 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 01:46:55.584338 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 01:46:55.602946 14092 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:46:55.604460 14092 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:46:55.612421 14092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e1c2fa7294c44dc812bcb858e4f1196: Generated new TSK 0
I20260810 01:46:55.613415 14092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:46:55.628733 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 13962
2026-08-10T01:46:55Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1751 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:46:55 dist-test-slave-kvfs krb5kdc[14121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:46:55 dist-test-slave-kvfs krb5kdc[14121](info): set up 2 sockets
Aug 10 01:46:55 dist-test-slave-kvfs krb5kdc[14121](info): commencing operation
krb5kdc: starting...
W20260810 01:46:56.954057 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.291s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:46:57 dist-test-slave-kvfs krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:46:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:46:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:46:57.105290 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:32923
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:32923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:46:57.241603 14145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:46:57.242188 14145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:46:57.242266 14145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:46:57.246742 14145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:46:57.246841 14145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:46:57.246878 14145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:46:57.246934 14145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:46:57.246971 14145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:46:57.254850 14145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:32923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:32923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:46:57.256450 14145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:46:57.257853 14145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:46:57.268671 14154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:57.268692 14152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:57.268677 14151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:57.269181 14145 server_base.cc:1034] running on GCE node
I20260810 01:46:57.269935 14145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:46:57.271368 14145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:46:57.272539 14145 hybrid_clock.cc:648] HybridClock initialized: now 1786326417272519 us; error 39 us; skew 500 ppm
Aug 10 01:46:57 dist-test-slave-kvfs krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326417, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:46:57.275905 14145 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:46:57.277123 14145 webserver.cc:460] Webserver started at http://127.9.77.190:40245/ using document root <none> and password file <none>
I20260810 01:46:57.277825 14145 fs_manager.cc:359] Metadata directory not provided
I20260810 01:46:57.277935 14145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:46:57.278223 14145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:46:57.280892 14145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "e8e68c91e65542b7805f4f0c1c626509"
format_stamp: "Formatted at 2026-08-10 01:46:57 on dist-test-slave-kvfs"
server_key: "e2a0449219673f0313556de799bcc80a"
server_key_iv: "0632d143d31dd8e7cf4fc0f2091c480e"
server_key_version: "encryptionkey@0"
I20260810 01:46:57.281610 14145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "e8e68c91e65542b7805f4f0c1c626509"
format_stamp: "Formatted at 2026-08-10 01:46:57 on dist-test-slave-kvfs"
server_key: "e2a0449219673f0313556de799bcc80a"
server_key_iv: "0632d143d31dd8e7cf4fc0f2091c480e"
server_key_version: "encryptionkey@0"
I20260810 01:46:57.286442 14145 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260810 01:46:57.289578 14162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:57.291218 14145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 01:46:57.291390 14145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "e8e68c91e65542b7805f4f0c1c626509"
format_stamp: "Formatted at 2026-08-10 01:46:57 on dist-test-slave-kvfs"
server_key: "e2a0449219673f0313556de799bcc80a"
server_key_iv: "0632d143d31dd8e7cf4fc0f2091c480e"
server_key_version: "encryptionkey@0"
I20260810 01:46:57.291536 14145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:46:57.312201 14145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:46:57.316094 14145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:46:57.316334 14145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:46:57.325373 14145 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:32923
I20260810 01:46:57.325752 14255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:32923 every 8 connection(s)
I20260810 01:46:57.326714 14145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:46:57.330746 14257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:46:57.336309 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 14145
I20260810 01:46:57.336449 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:46:57.336796 9526 external_mini_cluster.cc:1428] Setting key c88a6eb8334d1529397f47cdb396e220
I20260810 01:46:57.338825 14257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8e68c91e65542b7805f4f0c1c626509: Bootstrap starting.
I20260810 01:46:57.342216 14257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8e68c91e65542b7805f4f0c1c626509: Neither blocks nor log segments found. Creating new log.
I20260810 01:46:57.343371 14257 log.cc:824] T 00000000000000000000000000000000 P e8e68c91e65542b7805f4f0c1c626509: Log is configured to *not* fsync() on all Append() calls
I20260810 01:46:57.346067 14257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8e68c91e65542b7805f4f0c1c626509: No bootstrap required, opened a new log
Aug 10 01:46:57 dist-test-slave-kvfs krb5kdc[14121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:57.349818 14257 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8e68c91e65542b7805f4f0c1c626509 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e68c91e65542b7805f4f0c1c626509" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 32923 } }
I20260810 01:46:57.350076 14257 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8e68c91e65542b7805f4f0c1c626509 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:46:57.350152 14257 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8e68c91e65542b7805f4f0c1c626509 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8e68c91e65542b7805f4f0c1c626509, State: Initialized, Role: FOLLOWER
I20260810 01:46:57.350685 14257 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8e68c91e65542b7805f4f0c1c626509 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e68c91e65542b7805f4f0c1c626509" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 32923 } }
I20260810 01:46:57.350858 14257 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8e68c91e65542b7805f4f0c1c626509 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:46:57.350955 14257 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8e68c91e65542b7805f4f0c1c626509 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:46:57.351101 14257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8e68c91e65542b7805f4f0c1c626509 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:46:57.352173 14257 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8e68c91e65542b7805f4f0c1c626509 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e68c91e65542b7805f4f0c1c626509" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 32923 } }
I20260810 01:46:57.352576 14257 leader_election.cc:302] T 00000000000000000000000000000000 P e8e68c91e65542b7805f4f0c1c626509 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8e68c91e65542b7805f4f0c1c626509; no voters:
I20260810 01:46:57.352972 14257 leader_election.cc:288] T 00000000000000000000000000000000 P e8e68c91e65542b7805f4f0c1c626509 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:46:57.353106 14270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8e68c91e65542b7805f4f0c1c626509 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:46:57.353364 14270 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8e68c91e65542b7805f4f0c1c626509 [term 1 LEADER]: Becoming Leader. State: Replica: e8e68c91e65542b7805f4f0c1c626509, State: Running, Role: LEADER
I20260810 01:46:57.353761 14270 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8e68c91e65542b7805f4f0c1c626509 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e68c91e65542b7805f4f0c1c626509" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 32923 } }
I20260810 01:46:57.354552 14257 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8e68c91e65542b7805f4f0c1c626509 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:46:57.354699 14266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:57.338565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51118 (local address 127.9.77.190:32923)
0810 01:46:57.339084 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:57.339104 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:57.339152 (+ 48us) server_negotiation.cc:407] Connection header received
0810 01:46:57.339875 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:57.339895 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:57.340375 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:57.340708 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:57.341815 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:57.343034 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:57.344305 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:57.344845 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:57.344983 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:57.348132 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:57.348137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:57.348153 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:57.348182 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:57.351097 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:57.351593 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:57.351600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:57.351606 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:57.351734 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:57.352061 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:57.352064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:57.352066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:57.352591 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:57.352834 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:57.353102 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 01:46:57.353640 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":162,"threads_started":1}
I20260810 01:46:57.355707 14273 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e68c91e65542b7805f4f0c1c626509 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8e68c91e65542b7805f4f0c1c626509" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e68c91e65542b7805f4f0c1c626509" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 32923 } } }
I20260810 01:46:57.355867 14273 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e68c91e65542b7805f4f0c1c626509 [sys.catalog]: This master's current role is: LEADER
I20260810 01:46:57.356161 14273 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e68c91e65542b7805f4f0c1c626509 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8e68c91e65542b7805f4f0c1c626509. Latest consensus state: current_term: 1 leader_uuid: "e8e68c91e65542b7805f4f0c1c626509" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e68c91e65542b7805f4f0c1c626509" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 32923 } } }
I20260810 01:46:57.356251 14273 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e68c91e65542b7805f4f0c1c626509 [sys.catalog]: This master's current role is: LEADER
I20260810 01:46:57.357458 14284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:46:57.360826 14284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:46:57.367970 14284 catalog_manager.cc:1347] Generated new cluster ID: ef1bb6f5ddf546c2a8c857915b621c43
I20260810 01:46:57.368098 14284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:46:57.393613 14284 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:46:57.395314 14284 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:46:57.404616 14284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8e68c91e65542b7805f4f0c1c626509: Generated new TSK 0
I20260810 01:46:57.405692 14284 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.77.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
I20260810 01:46:57.441629 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:0
--local_ip_for_outbound_sockets=127.9.77.129
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:32923
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:46:57.599965 14306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:46:57.600477 14306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:46:57.600565 14306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:46:57.605742 14306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:46:57.605837 14306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:46:57.605937 14306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:46:57.612092 14306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:32923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:46:57.613970 14306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:46:57.615360 14306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:46:57.627672 14312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:57.627835 14306 server_base.cc:1034] running on GCE node
W20260810 01:46:57.627713 14313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:57.627718 14315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:57.628592 14306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:46:57.629385 14306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:46:57.630563 14306 hybrid_clock.cc:648] HybridClock initialized: now 1786326417630546 us; error 31 us; skew 500 ppm
Aug 10 01:46:57 dist-test-slave-kvfs krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326417, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:46:57.633818 14306 init.cc:377] Logged in from keytab as kudu/127.9.77.129@KRBTEST.COM (short username kudu)
I20260810 01:46:57.634940 14306 webserver.cc:460] Webserver started at http://127.9.77.129:37895/ using document root <none> and password file <none>
I20260810 01:46:57.635641 14306 fs_manager.cc:359] Metadata directory not provided
I20260810 01:46:57.635738 14306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:46:57.636005 14306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:46:57.638500 14306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/data/instance:
uuid: "b918684af19e4f7cb2259fe9f8569cb7"
format_stamp: "Formatted at 2026-08-10 01:46:57 on dist-test-slave-kvfs"
server_key: "726a578378c514071b75b3ce00405b20"
server_key_iv: "eb6e921307c4978d03f09836a57db08f"
server_key_version: "encryptionkey@0"
I20260810 01:46:57.639156 14306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance:
uuid: "b918684af19e4f7cb2259fe9f8569cb7"
format_stamp: "Formatted at 2026-08-10 01:46:57 on dist-test-slave-kvfs"
server_key: "726a578378c514071b75b3ce00405b20"
server_key_iv: "eb6e921307c4978d03f09836a57db08f"
server_key_version: "encryptionkey@0"
I20260810 01:46:57.643724 14306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260810 01:46:57.646629 14330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:57.648180 14306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:46:57.648320 14306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "b918684af19e4f7cb2259fe9f8569cb7"
format_stamp: "Formatted at 2026-08-10 01:46:57 on dist-test-slave-kvfs"
server_key: "726a578378c514071b75b3ce00405b20"
server_key_iv: "eb6e921307c4978d03f09836a57db08f"
server_key_version: "encryptionkey@0"
I20260810 01:46:57.648458 14306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:46:57.670047 14306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:46:57.674153 14306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:46:57.674422 14306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:46:57.675387 14306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:46:57.676712 14306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:46:57.676767 14306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:57.676819 14306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:46:57.676842 14306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:57.687775 14306 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:32963
I20260810 01:46:57.687799 14511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:32963 every 8 connection(s)
I20260810 01:46:57.689077 14306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:46:57.696221 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 14306
I20260810 01:46:57.696400 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance
I20260810 01:46:57.696794 9526 external_mini_cluster.cc:1428] Setting key 58407da952ef3e2d315f99e42a6a710a
Aug 10 01:46:57 dist-test-slave-kvfs krb5kdc[14121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326417, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:57.706326 14266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:57.691591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:52643 (local address 127.9.77.190:32923)
0810 01:46:57.691767 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:57.691772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:57.692940 (+ 1168us) server_negotiation.cc:407] Connection header received
0810 01:46:57.694518 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:57.694522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:57.694582 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:57.694666 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:57.696235 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:57.697034 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:57.698808 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:57.699077 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:57.699196 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:57.702186 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:57.702190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:57.702193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:57.702221 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:57.703864 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:57.704498 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:57.704501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:57.704503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:57.704612 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:57.704995 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:57.704999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:57.705001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:57.705238 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:57.705355 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:57.705961 (+ 606us) server_negotiation.cc:299] Negotiation successful
0810 01:46:57.706126 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 01:46:57.707417 14515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:57.692105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:32923 (local address 127.9.77.129:52643)
0810 01:46:57.692730 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:57.692783 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:57.694268 (+ 1485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:57.694889 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:57.694897 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:57.695327 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:46:57.696057 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:57.696084 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:57.697178 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:57.697182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:57.698674 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:57.698684 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:57.699244 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:57.699248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:57.699465 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:57.700279 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:46:57.700298 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:46:57.702005 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:46:57.704014 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:57.704044 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:57.704067 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:57.704378 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:57.704712 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:57.704716 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:57.704718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:57.704887 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:57.705355 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:46:57.705365 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:46:57.705683 (+ 318us) client_negotiation.cc:769] Sending connection context
0810 01:46:57.705903 (+ 220us) client_negotiation.cc:240] Negotiation successful
0810 01:46:57.706330 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":219,"threads_started":1}
I20260810 01:46:57.709087 14512 heartbeater.cc:347] Connected to a master server at 127.9.77.190:32923
I20260810 01:46:57.709451 14512 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.9.77.130@KRBTEST.COM; defaulting to no policy
I20260810 01:46:57.710242 14512 heartbeater.cc:511] Master 127.9.77.190:32923 requested a full tablet report, sending...
I20260810 01:46:57.712281 14192 ts_manager.cc:194] Registered new tserver with Master: b918684af19e4f7cb2259fe9f8569cb7 (127.9.77.129:32963)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.130@KRBTEST.COM" created.
I20260810 01:46:57.714047 14192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.129@KRBTEST.COM'} at 127.9.77.129:52643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
I20260810 01:46:57.732808 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:0
--local_ip_for_outbound_sockets=127.9.77.130
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:32923
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:46:57.870267 14518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:46:57.870611 14518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:46:57.870657 14518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:46:57.874936 14518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:46:57.875018 14518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:46:57.875118 14518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:46:57.881108 14518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:32923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:46:57.882546 14518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:46:57.883878 14518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:46:57.895584 14526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:57.895581 14525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:57.896114 14518 server_base.cc:1034] running on GCE node
W20260810 01:46:57.895584 14528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:57.896615 14518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:46:57.897421 14518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:46:57.898623 14518 hybrid_clock.cc:648] HybridClock initialized: now 1786326417898594 us; error 52 us; skew 500 ppm
Aug 10 01:46:57 dist-test-slave-kvfs krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326417, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:46:57.902024 14518 init.cc:377] Logged in from keytab as kudu/127.9.77.130@KRBTEST.COM (short username kudu)
I20260810 01:46:57.903158 14518 webserver.cc:460] Webserver started at http://127.9.77.130:32901/ using document root <none> and password file <none>
I20260810 01:46:57.903904 14518 fs_manager.cc:359] Metadata directory not provided
I20260810 01:46:57.904013 14518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:46:57.904350 14518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:46:57.907055 14518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/data/instance:
uuid: "b842b583752a44638968d099ad754fb2"
format_stamp: "Formatted at 2026-08-10 01:46:57 on dist-test-slave-kvfs"
server_key: "6d7bb19d6155fc927caebf2abdcdc22f"
server_key_iv: "f678c4398e2409b4c397e613328c7fbc"
server_key_version: "encryptionkey@0"
I20260810 01:46:57.907815 14518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance:
uuid: "b842b583752a44638968d099ad754fb2"
format_stamp: "Formatted at 2026-08-10 01:46:57 on dist-test-slave-kvfs"
server_key: "6d7bb19d6155fc927caebf2abdcdc22f"
server_key_iv: "f678c4398e2409b4c397e613328c7fbc"
server_key_version: "encryptionkey@0"
I20260810 01:46:57.912741 14518 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 01:46:57.915735 14538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:57.917428 14518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:46:57.917594 14518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "b842b583752a44638968d099ad754fb2"
format_stamp: "Formatted at 2026-08-10 01:46:57 on dist-test-slave-kvfs"
server_key: "6d7bb19d6155fc927caebf2abdcdc22f"
server_key_iv: "f678c4398e2409b4c397e613328c7fbc"
server_key_version: "encryptionkey@0"
I20260810 01:46:57.917738 14518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:46:57.952365 14518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:46:57.956980 14518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:46:57.957247 14518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:46:57.958069 14518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:46:57.959199 14518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:46:57.959488 14518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:57.959586 14518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:46:57.959647 14518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:57.971242 14518 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:43523
I20260810 01:46:57.971271 14706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:43523 every 8 connection(s)
I20260810 01:46:57.972601 14518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:46:57.973466 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 14518
I20260810 01:46:57.973601 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance
I20260810 01:46:57.973982 9526 external_mini_cluster.cc:1428] Setting key 47519bb74b7fd6b85684950097e7e805
Aug 10 01:46:57 dist-test-slave-kvfs krb5kdc[14121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326417, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.77.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.131@KRBTEST.COM" created.
I20260810 01:46:57.992426 14266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:57.975280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:33107 (local address 127.9.77.190:32923)
0810 01:46:57.975443 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:57.975447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:57.977025 (+ 1578us) server_negotiation.cc:407] Connection header received
0810 01:46:57.979578 (+ 2553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:57.979584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:57.979671 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:57.979821 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:57.981444 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:57.982401 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:57.983894 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:57.984341 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:57.984486 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:57.987885 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:57.987890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:57.987894 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:57.987935 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:57.989734 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:57.990349 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:57.990353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:57.990355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:57.990459 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:57.990908 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:57.990912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:57.990913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:57.991175 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:57.991297 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:57.992053 (+ 756us) server_negotiation.cc:299] Negotiation successful
0810 01:46:57.992219 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 01:46:57.993705 14709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:57.975988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:32923 (local address 127.9.77.130:33107)
0810 01:46:57.976799 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:57.976875 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:57.979197 (+ 2322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:57.980097 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:57.980106 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:57.980598 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:46:57.981260 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:57.981278 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:57.982566 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:57.982570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:57.983741 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:57.983761 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:57.984528 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:57.984532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:57.984721 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:57.985607 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:46:57.985627 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:46:57.987662 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:46:57.989875 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:57.989884 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:57.989913 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:57.990216 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:57.990567 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:57.990570 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:57.990572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:57.990809 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:57.991300 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:46:57.991310 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:46:57.991683 (+ 373us) client_negotiation.cc:769] Sending connection context
0810 01:46:57.991977 (+ 294us) client_negotiation.cc:240] Negotiation successful
0810 01:46:57.992443 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":218,"threads_started":1}
I20260810 01:46:57.995374 14707 heartbeater.cc:347] Connected to a master server at 127.9.77.190:32923
I20260810 01:46:57.995752 14707 heartbeater.cc:464] Registering TS with master...
I20260810 01:46:57.996534 14707 heartbeater.cc:511] Master 127.9.77.190:32923 requested a full tablet report, sending...
I20260810 01:46:57.997967 14192 ts_manager.cc:194] Registered new tserver with Master: b842b583752a44638968d099ad754fb2 (127.9.77.130:43523)
I20260810 01:46:57.998719 14192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.130@KRBTEST.COM'} at 127.9.77.130:33107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
I20260810 01:46:58.011821 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:0
--local_ip_for_outbound_sockets=127.9.77.131
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:32923
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:44915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:46:58.149307 14712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:46:58.149704 14712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:46:58.149788 14712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:46:58.154212 14712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:46:58.154302 14712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:46:58.154399 14712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:46:58.160386 14712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:44915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:32923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:46:58.161835 14712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:46:58.163187 14712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:46:58.175022 14725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:58.175101 14712 server_base.cc:1034] running on GCE node
W20260810 01:46:58.175040 14722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:58.175040 14730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:58.175725 14712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:46:58.176544 14712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:46:58.177743 14712 hybrid_clock.cc:648] HybridClock initialized: now 1786326418177722 us; error 36 us; skew 500 ppm
Aug 10 01:46:58 dist-test-slave-kvfs krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326418, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:46:58.180984 14712 init.cc:377] Logged in from keytab as kudu/127.9.77.131@KRBTEST.COM (short username kudu)
I20260810 01:46:58.182051 14712 webserver.cc:460] Webserver started at http://127.9.77.131:39845/ using document root <none> and password file <none>
I20260810 01:46:58.182749 14712 fs_manager.cc:359] Metadata directory not provided
I20260810 01:46:58.182847 14712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:46:58.183128 14712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:46:58.185708 14712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/data/instance:
uuid: "7ebcf9b55cb849e5b20f0075d5031132"
format_stamp: "Formatted at 2026-08-10 01:46:58 on dist-test-slave-kvfs"
server_key: "e2e2183dbec6b3f35ef916bd5c81fafa"
server_key_iv: "f1cf01a395c23f1ceb03819783f0f5dc"
server_key_version: "encryptionkey@0"
I20260810 01:46:58.186376 14712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance:
uuid: "7ebcf9b55cb849e5b20f0075d5031132"
format_stamp: "Formatted at 2026-08-10 01:46:58 on dist-test-slave-kvfs"
server_key: "e2e2183dbec6b3f35ef916bd5c81fafa"
server_key_iv: "f1cf01a395c23f1ceb03819783f0f5dc"
server_key_version: "encryptionkey@0"
I20260810 01:46:58.191099 14712 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 01:46:58.194124 14739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:58.195756 14712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 01:46:58.195909 14712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "7ebcf9b55cb849e5b20f0075d5031132"
format_stamp: "Formatted at 2026-08-10 01:46:58 on dist-test-slave-kvfs"
server_key: "e2e2183dbec6b3f35ef916bd5c81fafa"
server_key_iv: "f1cf01a395c23f1ceb03819783f0f5dc"
server_key_version: "encryptionkey@0"
I20260810 01:46:58.196097 14712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:46:58.210999 14712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:46:58.214588 14712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:46:58.214819 14712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:46:58.215545 14712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:46:58.216567 14712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:46:58.216642 14712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:58.216722 14712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:46:58.216775 14712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:58.227375 14712 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:38053
I20260810 01:46:58.227416 14909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:38053 every 8 connection(s)
I20260810 01:46:58.228555 14712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:46:58.232101 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 14712
I20260810 01:46:58.232250 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance
I20260810 01:46:58.232633 9526 external_mini_cluster.cc:1428] Setting key c8c8321794ec99d974d33c9776abd0d0
Aug 10 01:46:58 dist-test-slave-kvfs krb5kdc[14121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326418, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:58.244532 14266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:58.230808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:37363 (local address 127.9.77.190:32923)
0810 01:46:58.230990 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:58.230995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:58.231879 (+ 884us) server_negotiation.cc:407] Connection header received
0810 01:46:58.233834 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:58.233838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:58.233906 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:58.233997 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:58.235539 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:58.236255 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:58.237479 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:58.237708 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:58.237795 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:58.240552 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:58.240556 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:58.240558 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:58.240585 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:58.242307 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:58.242891 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:58.242893 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:58.242895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:58.242977 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:58.243308 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:58.243310 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:58.243311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:58.243522 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:58.243600 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:58.244243 (+ 643us) server_negotiation.cc:299] Negotiation successful
0810 01:46:58.244375 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 01:46:58.245738 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:58.231181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:32923 (local address 127.9.77.131:37363)
0810 01:46:58.231699 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0810 01:46:58.231749 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 01:46:58.233544 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:46:58.234243 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:46:58.234251 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:46:58.234680 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:46:58.235376 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:58.235394 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:58.236383 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:58.236387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:58.237359 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:46:58.237368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:46:58.237847 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:46:58.237850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:46:58.238036 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:58.238679 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:46:58.238697 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:46:58.240390 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:46:58.242452 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:58.242458 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:58.242473 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:58.242799 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:58.243067 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:46:58.243070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:46:58.243071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:46:58.243223 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:46:58.243621 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:46:58.243627 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:46:58.243936 (+ 309us) client_negotiation.cc:769] Sending connection context
0810 01:46:58.244199 (+ 263us) client_negotiation.cc:240] Negotiation successful
0810 01:46:58.244568 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":126,"threads_started":1}
I20260810 01:46:58.247241 14910 heartbeater.cc:347] Connected to a master server at 127.9.77.190:32923
I20260810 01:46:58.247574 14910 heartbeater.cc:464] Registering TS with master...
I20260810 01:46:58.248242 14910 heartbeater.cc:511] Master 127.9.77.190:32923 requested a full tablet report, sending...
I20260810 01:46:58.249400 14192 ts_manager.cc:194] Registered new tserver with Master: 7ebcf9b55cb849e5b20f0075d5031132 (127.9.77.131:38053)
I20260810 01:46:58.250046 14192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.131@KRBTEST.COM'} at 127.9.77.131:37363
I20260810 01:46:58.259406 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 01:46:58 dist-test-slave-kvfs krb5kdc[14121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326418, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 01:46:58 dist-test-slave-kvfs krb5kdc[14121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326418, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:46:58.290349 14266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:46:58.280908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51134 (local address 127.9.77.190:32923)
0810 01:46:58.281122 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 01:46:58.281129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:46:58.281279 (+ 150us) server_negotiation.cc:407] Connection header received
0810 01:46:58.281501 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:46:58.281505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:46:58.281562 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:46:58.281657 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:46:58.282452 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:58.283061 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:58.283982 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:46:58.284282 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:46:58.284403 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:46:58.287074 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:46:58.287078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:46:58.287080 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:46:58.287108 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:46:58.288503 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:58.288933 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:58.288937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:58.288939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:58.289030 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:46:58.289316 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:46:58.289320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:46:58.289322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:46:58.289594 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:46:58.289758 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 01:46:58.290003 (+ 245us) server_negotiation.cc:299] Negotiation successful
0810 01:46:58.290142 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260810 01:46:58.290820 14192 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51134
I20260810 01:46:58.292585 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 14306
I20260810 01:46:58.313257 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 14518
I20260810 01:46:58.330286 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 14712
I20260810 01:46:58.349506 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 14145
2026-08-10T01:46:58Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2724 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:46:58 dist-test-slave-kvfs krb5kdc[14928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:46:58 dist-test-slave-kvfs krb5kdc[14928](info): set up 2 sockets
Aug 10 01:46:58 dist-test-slave-kvfs krb5kdc[14928](info): commencing operation
krb5kdc: starting...
W20260810 01:46:59.647028 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.259s user 0.002s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:46:59 dist-test-slave-kvfs krb5kdc[14928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:46:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:46:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:46:59.796742 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:41633
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:41129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:41633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:46:59.931325 14948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:46:59.931730 14948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:46:59.931800 14948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:46:59.935950 14948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:46:59.936079 14948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:46:59.936120 14948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:46:59.936182 14948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:46:59.936221 14948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:46:59.942015 14948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:41129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:41633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:41633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:46:59.943439 14948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:46:59.944793 14948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:46:59.955422 14960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:59.955443 14955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:46:59.955416 14956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:46:59.955876 14948 server_base.cc:1034] running on GCE node
I20260810 01:46:59.956667 14948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:46:59.958029 14948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:46:59.959249 14948 hybrid_clock.cc:648] HybridClock initialized: now 1786326419959213 us; error 47 us; skew 500 ppm
Aug 10 01:46:59 dist-test-slave-kvfs krb5kdc[14928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326419, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:46:59.962767 14948 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:46:59.964128 14948 webserver.cc:460] Webserver started at http://127.9.77.190:33733/ using document root <none> and password file <none>
I20260810 01:46:59.964900 14948 fs_manager.cc:359] Metadata directory not provided
I20260810 01:46:59.965014 14948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:46:59.965313 14948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:46:59.968009 14948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "a00e4badc55c4f97853c9a78c670eb21"
format_stamp: "Formatted at 2026-08-10 01:46:59 on dist-test-slave-kvfs"
server_key: "bfa17663271d9a51af9bcad556208c35"
server_key_iv: "7615c5fa2ed6af79ba074bdf5f2e6132"
server_key_version: "encryptionkey@0"
I20260810 01:46:59.968835 14948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "a00e4badc55c4f97853c9a78c670eb21"
format_stamp: "Formatted at 2026-08-10 01:46:59 on dist-test-slave-kvfs"
server_key: "bfa17663271d9a51af9bcad556208c35"
server_key_iv: "7615c5fa2ed6af79ba074bdf5f2e6132"
server_key_version: "encryptionkey@0"
I20260810 01:46:59.973592 14948 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 01:46:59.976706 14974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:46:59.978304 14948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:46:59.978492 14948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "a00e4badc55c4f97853c9a78c670eb21"
format_stamp: "Formatted at 2026-08-10 01:46:59 on dist-test-slave-kvfs"
server_key: "bfa17663271d9a51af9bcad556208c35"
server_key_iv: "7615c5fa2ed6af79ba074bdf5f2e6132"
server_key_version: "encryptionkey@0"
I20260810 01:46:59.978631 14948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:00.010349 14948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:00.014568 14948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:00.014946 14948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:00.024673 14948 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:41633
I20260810 01:47:00.024700 15059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:41633 every 8 connection(s)
I20260810 01:47:00.025979 14948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:47:00.026569 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 14948
I20260810 01:47:00.026844 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:47:00.027256 9526 external_mini_cluster.cc:1428] Setting key 958b5c490d37b07b85b1e0ff7c0aa61f
I20260810 01:47:00.030239 15060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:00.037528 15060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a00e4badc55c4f97853c9a78c670eb21: Bootstrap starting.
Aug 10 01:47:00 dist-test-slave-kvfs krb5kdc[14928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:47:00.040880 15060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a00e4badc55c4f97853c9a78c670eb21: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:00.041849 15060 log.cc:824] T 00000000000000000000000000000000 P a00e4badc55c4f97853c9a78c670eb21: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:00.044597 15060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a00e4badc55c4f97853c9a78c670eb21: No bootstrap required, opened a new log
I20260810 01:47:00.044891 15065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:00.029172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37488 (local address 127.9.77.190:41633)
0810 01:47:00.029753 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:00.029773 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:00.029820 (+ 47us) server_negotiation.cc:407] Connection header received
0810 01:47:00.030715 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:00.030737 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:00.031137 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:00.031549 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:00.032604 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:00.033791 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:00.034779 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:00.035333 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:00.035539 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:00.038759 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:00.038764 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:00.038780 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:00.038815 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:00.041041 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:00.041614 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:00.041620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:00.041626 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:00.041760 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:00.042116 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:00.042132 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:00.042133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:00.042816 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:00.043096 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:00.043452 (+ 356us) server_negotiation.cc:299] Negotiation successful
0810 01:47:00.043824 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":174,"threads_started":1}
I20260810 01:47:00.049660 15060 raft_consensus.cc:348] T 00000000000000000000000000000000 P a00e4badc55c4f97853c9a78c670eb21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00e4badc55c4f97853c9a78c670eb21" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 41633 } }
I20260810 01:47:00.049983 15060 raft_consensus.cc:374] T 00000000000000000000000000000000 P a00e4badc55c4f97853c9a78c670eb21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:00.050079 15060 raft_consensus.cc:729] T 00000000000000000000000000000000 P a00e4badc55c4f97853c9a78c670eb21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a00e4badc55c4f97853c9a78c670eb21, State: Initialized, Role: FOLLOWER
I20260810 01:47:00.050684 15060 consensus_queue.cc:260] T 00000000000000000000000000000000 P a00e4badc55c4f97853c9a78c670eb21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00e4badc55c4f97853c9a78c670eb21" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 41633 } }
I20260810 01:47:00.050915 15060 raft_consensus.cc:388] T 00000000000000000000000000000000 P a00e4badc55c4f97853c9a78c670eb21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:47:00.051021 15060 raft_consensus.cc:482] T 00000000000000000000000000000000 P a00e4badc55c4f97853c9a78c670eb21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:47:00.051165 15060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a00e4badc55c4f97853c9a78c670eb21 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:00.052462 15060 raft_consensus.cc:504] T 00000000000000000000000000000000 P a00e4badc55c4f97853c9a78c670eb21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00e4badc55c4f97853c9a78c670eb21" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 41633 } }
I20260810 01:47:00.052920 15060 leader_election.cc:302] T 00000000000000000000000000000000 P a00e4badc55c4f97853c9a78c670eb21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a00e4badc55c4f97853c9a78c670eb21; no voters:
I20260810 01:47:00.053324 15060 leader_election.cc:288] T 00000000000000000000000000000000 P a00e4badc55c4f97853c9a78c670eb21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:47:00.053543 15068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a00e4badc55c4f97853c9a78c670eb21 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:47:00.053845 15068 raft_consensus.cc:686] T 00000000000000000000000000000000 P a00e4badc55c4f97853c9a78c670eb21 [term 1 LEADER]: Becoming Leader. State: Replica: a00e4badc55c4f97853c9a78c670eb21, State: Running, Role: LEADER
I20260810 01:47:00.054241 15068 consensus_queue.cc:237] T 00000000000000000000000000000000 P a00e4badc55c4f97853c9a78c670eb21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00e4badc55c4f97853c9a78c670eb21" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 41633 } }
I20260810 01:47:00.055018 15060 sys_catalog.cc:564] T 00000000000000000000000000000000 P a00e4badc55c4f97853c9a78c670eb21 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:47:00.056439 15071 sys_catalog.cc:455] T 00000000000000000000000000000000 P a00e4badc55c4f97853c9a78c670eb21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a00e4badc55c4f97853c9a78c670eb21. Latest consensus state: current_term: 1 leader_uuid: "a00e4badc55c4f97853c9a78c670eb21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00e4badc55c4f97853c9a78c670eb21" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 41633 } } }
I20260810 01:47:00.056434 15069 sys_catalog.cc:455] T 00000000000000000000000000000000 P a00e4badc55c4f97853c9a78c670eb21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a00e4badc55c4f97853c9a78c670eb21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a00e4badc55c4f97853c9a78c670eb21" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 41633 } } }
I20260810 01:47:00.056596 15071 sys_catalog.cc:458] T 00000000000000000000000000000000 P a00e4badc55c4f97853c9a78c670eb21 [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:00.056596 15069 sys_catalog.cc:458] T 00000000000000000000000000000000 P a00e4badc55c4f97853c9a78c670eb21 [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:00.057037 15083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:47:00.061008 15083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:47:00.068065 15083 catalog_manager.cc:1347] Generated new cluster ID: 938389824727490aaff29d2a8aaf9103
I20260810 01:47:00.068162 15083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:47:00.081712 15083 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:47:00.083106 15083 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:47:00.099006 15083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a00e4badc55c4f97853c9a78c670eb21: Generated new TSK 0
I20260810 01:47:00.100234 15083 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.77.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
I20260810 01:47:00.139505 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:0
--local_ip_for_outbound_sockets=127.9.77.129
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:41633
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:41129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:00.274892 15103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:00.275241 15103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:00.275288 15103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:00.279515 15103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:00.279585 15103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:00.279646 15103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:47:00.285414 15103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:41129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:41633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:00.286721 15103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:00.288127 15103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:00.299908 15120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:00.299922 15111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:00.299966 15114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:00.300237 15103 server_base.cc:1034] running on GCE node
I20260810 01:47:00.300935 15103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:00.301653 15103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:00.302860 15103 hybrid_clock.cc:648] HybridClock initialized: now 1786326420302836 us; error 28 us; skew 500 ppm
Aug 10 01:47:00 dist-test-slave-kvfs krb5kdc[14928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326420, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:00.306635 15103 init.cc:377] Logged in from keytab as kudu/127.9.77.129@KRBTEST.COM (short username kudu)
I20260810 01:47:00.307813 15103 webserver.cc:460] Webserver started at http://127.9.77.129:43357/ using document root <none> and password file <none>
I20260810 01:47:00.308581 15103 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:00.308679 15103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:00.308944 15103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:00.311458 15103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-0/data/instance:
uuid: "7a2a5f20f95144a69a67ffe9aca47b71"
format_stamp: "Formatted at 2026-08-10 01:47:00 on dist-test-slave-kvfs"
server_key: "8dbce9d138f1cf45893c5bcc67bf9f88"
server_key_iv: "f2c58d632bc485414800573af2361048"
server_key_version: "encryptionkey@0"
I20260810 01:47:00.312192 15103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance:
uuid: "7a2a5f20f95144a69a67ffe9aca47b71"
format_stamp: "Formatted at 2026-08-10 01:47:00 on dist-test-slave-kvfs"
server_key: "8dbce9d138f1cf45893c5bcc67bf9f88"
server_key_iv: "f2c58d632bc485414800573af2361048"
server_key_version: "encryptionkey@0"
I20260810 01:47:00.317165 15103 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 01:47:00.320005 15129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:00.321489 15103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:47:00.321640 15103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "7a2a5f20f95144a69a67ffe9aca47b71"
format_stamp: "Formatted at 2026-08-10 01:47:00 on dist-test-slave-kvfs"
server_key: "8dbce9d138f1cf45893c5bcc67bf9f88"
server_key_iv: "f2c58d632bc485414800573af2361048"
server_key_version: "encryptionkey@0"
I20260810 01:47:00.321768 15103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:00.348578 15103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:00.352771 15103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:00.353025 15103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:00.353797 15103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:00.354903 15103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:00.354979 15103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:00.355052 15103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:00.355077 15103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:00.366148 15103 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:37109
I20260810 01:47:00.366173 15305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:37109 every 8 connection(s)
I20260810 01:47:00.367312 15103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:47:00.369228 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 15103
I20260810 01:47:00.369397 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance
I20260810 01:47:00.369850 9526 external_mini_cluster.cc:1428] Setting key a796c3fb12dbe56fa31671e64d95b5a2
Aug 10 01:47:00 dist-test-slave-kvfs krb5kdc[14928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326420, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.77.130@KRBTEST.COM; defaulting to no policy
I20260810 01:47:00.385254 15065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:00.369815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:39985 (local address 127.9.77.190:41633)
0810 01:47:00.370034 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:00.370040 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:00.371328 (+ 1288us) server_negotiation.cc:407] Connection header received
0810 01:47:00.373197 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:00.373203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:00.373276 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:00.373374 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:00.374878 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:00.375486 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:00.376852 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:00.377118 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:00.377239 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:00.380643 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:00.380646 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:00.380648 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:00.380680 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:00.382443 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:00.383085 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:00.383090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:00.383092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:00.383191 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:00.383657 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:00.383659 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:00.383660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:00.383900 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:00.383987 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:00.384910 (+ 923us) server_negotiation.cc:299] Negotiation successful
0810 01:47:00.385080 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 01:47:00.386447 15309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:00.370401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:41633 (local address 127.9.77.129:39985)
0810 01:47:00.371118 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:00.371188 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:00.372933 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:00.373591 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:00.373598 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:00.374032 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:00.374716 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:00.374733 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:00.375628 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:00.375632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:00.376704 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:00.376715 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:00.377291 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:00.377297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:00.377508 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:00.378299 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:00.378318 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:00.380418 (+ 2100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:00.382607 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:00.382616 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:00.382646 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:00.382955 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:00.383293 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:00.383297 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:00.383299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:00.383486 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:00.384075 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:00.384087 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:00.384484 (+ 397us) client_negotiation.cc:769] Sending connection context
0810 01:47:00.384750 (+ 266us) client_negotiation.cc:240] Negotiation successful
0810 01:47:00.385127 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":201,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.130@KRBTEST.COM" created.
I20260810 01:47:00.388185 15306 heartbeater.cc:347] Connected to a master server at 127.9.77.190:41633
I20260810 01:47:00.388540 15306 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:00.389350 15306 heartbeater.cc:511] Master 127.9.77.190:41633 requested a full tablet report, sending...
I20260810 01:47:00.391490 14996 ts_manager.cc:194] Registered new tserver with Master: 7a2a5f20f95144a69a67ffe9aca47b71 (127.9.77.129:37109)
I20260810 01:47:00.392894 14996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.129@KRBTEST.COM'} at 127.9.77.129:39985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
I20260810 01:47:00.408090 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:0
--local_ip_for_outbound_sockets=127.9.77.130
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:41633
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:41129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:00.542253 15314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:00.542670 15314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:00.542758 15314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:00.547022 15314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:00.547148 15314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:00.547251 15314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:47:00.553297 15314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:41129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:41633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:00.554888 15314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:00.556367 15314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:00.568324 15322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:00.568493 15314 server_base.cc:1034] running on GCE node
W20260810 01:47:00.568325 15324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:00.568325 15326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:00.569252 15314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:00.570122 15314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:00.571326 15314 hybrid_clock.cc:648] HybridClock initialized: now 1786326420571305 us; error 33 us; skew 500 ppm
Aug 10 01:47:00 dist-test-slave-kvfs krb5kdc[14928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326420, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:00.578971 15314 init.cc:377] Logged in from keytab as kudu/127.9.77.130@KRBTEST.COM (short username kudu)
I20260810 01:47:00.580474 15314 webserver.cc:460] Webserver started at http://127.9.77.130:41497/ using document root <none> and password file <none>
I20260810 01:47:00.581488 15314 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:00.581593 15314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:00.581938 15314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:00.584864 15314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-1/data/instance:
uuid: "590fcbf82b934f18ae78ada551e72a8c"
format_stamp: "Formatted at 2026-08-10 01:47:00 on dist-test-slave-kvfs"
server_key: "3f8927b68e030c5c7da7171172d8da9e"
server_key_iv: "929ac79ac158e664af960af97ea2d8fc"
server_key_version: "encryptionkey@0"
I20260810 01:47:00.585496 15314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance:
uuid: "590fcbf82b934f18ae78ada551e72a8c"
format_stamp: "Formatted at 2026-08-10 01:47:00 on dist-test-slave-kvfs"
server_key: "3f8927b68e030c5c7da7171172d8da9e"
server_key_iv: "929ac79ac158e664af960af97ea2d8fc"
server_key_version: "encryptionkey@0"
I20260810 01:47:00.590402 15314 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 01:47:00.593629 15337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:00.595355 15314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 01:47:00.595499 15314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "590fcbf82b934f18ae78ada551e72a8c"
format_stamp: "Formatted at 2026-08-10 01:47:00 on dist-test-slave-kvfs"
server_key: "3f8927b68e030c5c7da7171172d8da9e"
server_key_iv: "929ac79ac158e664af960af97ea2d8fc"
server_key_version: "encryptionkey@0"
I20260810 01:47:00.595674 15314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:00.607838 15314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:00.611514 15314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:00.611703 15314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:00.612473 15314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:00.613461 15314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:00.613510 15314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:00.613555 15314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:00.613570 15314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:00.624562 15314 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:42533
I20260810 01:47:00.624634 15523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:42533 every 8 connection(s)
I20260810 01:47:00.625806 15314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:47:00.627696 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 15314
I20260810 01:47:00.627842 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance
I20260810 01:47:00.628273 9526 external_mini_cluster.cc:1428] Setting key 15a30d9ca4292676578d3d3b58f2f0b4
Aug 10 01:47:00 dist-test-slave-kvfs krb5kdc[14928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326420, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.77.131@KRBTEST.COM; defaulting to no policy
I20260810 01:47:00.644001 15065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:00.628483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:57663 (local address 127.9.77.190:41633)
0810 01:47:00.628686 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:00.628692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:00.629893 (+ 1201us) server_negotiation.cc:407] Connection header received
0810 01:47:00.631533 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:00.631537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:00.631598 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:00.631682 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:00.633297 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:00.633926 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:00.635834 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:00.636198 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:00.636309 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:00.639847 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:00.639850 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:00.639853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:00.639884 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:00.641561 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:00.642143 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:00.642149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:00.642151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:00.642277 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:00.642656 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:00.642661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:00.642663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:00.642898 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:00.643017 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:00.643634 (+ 617us) server_negotiation.cc:299] Negotiation successful
0810 01:47:00.643793 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.131@KRBTEST.COM" created.
I20260810 01:47:00.645324 15528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:00.629091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:41633 (local address 127.9.77.130:57663)
0810 01:47:00.629719 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:00.629784 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:00.631290 (+ 1506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:00.631895 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:00.631903 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:00.632403 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:00.633097 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:00.633123 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:00.634065 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:00.634071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:00.635656 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:00.635671 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:00.636388 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:00.636395 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:00.636655 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:00.637581 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:00.637614 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:00.639629 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:00.641698 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:00.641706 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:00.641727 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:00.642027 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:00.642379 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:00.642383 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:00.642385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:00.642553 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:00.643013 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:00.643023 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:00.643348 (+ 325us) client_negotiation.cc:769] Sending connection context
0810 01:47:00.643577 (+ 229us) client_negotiation.cc:240] Negotiation successful
0810 01:47:00.644013 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":171,"threads_started":1}
I20260810 01:47:00.646914 15525 heartbeater.cc:347] Connected to a master server at 127.9.77.190:41633
I20260810 01:47:00.647267 15525 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:00.647976 15525 heartbeater.cc:511] Master 127.9.77.190:41633 requested a full tablet report, sending...
I20260810 01:47:00.649600 14996 ts_manager.cc:194] Registered new tserver with Master: 590fcbf82b934f18ae78ada551e72a8c (127.9.77.130:42533)
I20260810 01:47:00.650326 14996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.130@KRBTEST.COM'} at 127.9.77.130:57663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
I20260810 01:47:00.667995 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:0
--local_ip_for_outbound_sockets=127.9.77.131
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:41633
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:41129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:00.805040 15531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:00.805435 15531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:00.805524 15531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:00.810031 15531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:00.810105 15531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:00.810168 15531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:47:00.815976 15531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:41129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:41633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:00.817655 15531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:00.819023 15531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:00.831967 15539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:00.831959 15536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:00.832243 15531 server_base.cc:1034] running on GCE node
W20260810 01:47:00.832068 15537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:00.832916 15531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:00.833784 15531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:00.834959 15531 hybrid_clock.cc:648] HybridClock initialized: now 1786326420834940 us; error 32 us; skew 500 ppm
Aug 10 01:47:00 dist-test-slave-kvfs krb5kdc[14928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326420, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:00.838757 15531 init.cc:377] Logged in from keytab as kudu/127.9.77.131@KRBTEST.COM (short username kudu)
I20260810 01:47:00.839958 15531 webserver.cc:460] Webserver started at http://127.9.77.131:38497/ using document root <none> and password file <none>
I20260810 01:47:00.840703 15531 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:00.840775 15531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:00.840993 15531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:00.843487 15531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-2/data/instance:
uuid: "a13c6606867346db81872850bf2de970"
format_stamp: "Formatted at 2026-08-10 01:47:00 on dist-test-slave-kvfs"
server_key: "70ce88e42f58219ba5a31beedf1db6ea"
server_key_iv: "2788e1cff13874cdc8bea1eac012ae0b"
server_key_version: "encryptionkey@0"
I20260810 01:47:00.844177 15531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance:
uuid: "a13c6606867346db81872850bf2de970"
format_stamp: "Formatted at 2026-08-10 01:47:00 on dist-test-slave-kvfs"
server_key: "70ce88e42f58219ba5a31beedf1db6ea"
server_key_iv: "2788e1cff13874cdc8bea1eac012ae0b"
server_key_version: "encryptionkey@0"
I20260810 01:47:00.848798 15531 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 01:47:00.851750 15554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:00.853368 15531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:47:00.853515 15531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "a13c6606867346db81872850bf2de970"
format_stamp: "Formatted at 2026-08-10 01:47:00 on dist-test-slave-kvfs"
server_key: "70ce88e42f58219ba5a31beedf1db6ea"
server_key_iv: "2788e1cff13874cdc8bea1eac012ae0b"
server_key_version: "encryptionkey@0"
I20260810 01:47:00.853649 15531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:00.868788 15531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:00.872401 15531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:00.872644 15531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:00.873415 15531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:00.874472 15531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:00.874524 15531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:00.874572 15531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:00.874630 15531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:00.886194 15531 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:40197
I20260810 01:47:00.886212 15733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:40197 every 8 connection(s)
I20260810 01:47:00.887532 15531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:47:00.888167 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 15531
I20260810 01:47:00.888317 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance
I20260810 01:47:00.888757 9526 external_mini_cluster.cc:1428] Setting key 5ae4a2ce05720bb18f8931c4f5379cc0
Aug 10 01:47:00 dist-test-slave-kvfs krb5kdc[14928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326420, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:47:00.904996 15065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:00.890283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:49065 (local address 127.9.77.190:41633)
0810 01:47:00.890473 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:00.890480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:00.891800 (+ 1320us) server_negotiation.cc:407] Connection header received
0810 01:47:00.893569 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:00.893572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:00.893630 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:00.893710 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:00.895254 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:00.895879 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:00.897250 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:00.897514 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:00.897625 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:00.900751 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:00.900754 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:00.900757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:00.900788 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:00.902486 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:00.903097 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:00.903101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:00.903103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:00.903215 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:00.903662 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:00.903665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:00.903666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:00.903893 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:00.904015 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:00.904643 (+ 628us) server_negotiation.cc:299] Negotiation successful
0810 01:47:00.904800 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 01:47:00.906201 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:00.890983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:41633 (local address 127.9.77.131:49065)
0810 01:47:00.891622 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:00.891689 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:00.893318 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:00.893919 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:00.893928 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:00.894378 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:00.895031 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:00.895047 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:00.896011 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:00.896014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:00.897099 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:00.897108 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:00.897707 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:00.897710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:00.897951 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:00.898740 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:00.898759 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:00.900530 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:00.902616 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:00.902621 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:00.902638 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:00.902977 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:00.903326 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:00.903329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:00.903331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:00.903542 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:00.904009 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:00.904017 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:00.904363 (+ 346us) client_negotiation.cc:769] Sending connection context
0810 01:47:00.904583 (+ 220us) client_negotiation.cc:240] Negotiation successful
0810 01:47:00.905017 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":184,"threads_started":1}
I20260810 01:47:00.907625 15734 heartbeater.cc:347] Connected to a master server at 127.9.77.190:41633
I20260810 01:47:00.908135 15734 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:00.908871 15734 heartbeater.cc:511] Master 127.9.77.190:41633 requested a full tablet report, sending...
I20260810 01:47:00.910394 14996 ts_manager.cc:194] Registered new tserver with Master: a13c6606867346db81872850bf2de970 (127.9.77.131:40197)
I20260810 01:47:00.911182 14996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.131@KRBTEST.COM'} at 127.9.77.131:49065
I20260810 01:47:00.916713 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 01:47:00 dist-test-slave-kvfs krb5kdc[14928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 01:47:00 dist-test-slave-kvfs krb5kdc[14928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.129@KRBTEST.COM
I20260810 01:47:00.953121 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:00.940919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37502 (local address 127.9.77.129:37109)
0810 01:47:00.941290 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:00.941301 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:00.941336 (+ 35us) server_negotiation.cc:407] Connection header received
0810 01:47:00.941419 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:00.941426 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:00.941686 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:00.941846 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:00.942674 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:00.943578 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:00.944641 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:00.945134 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:00.945233 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:00.948521 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:00.948538 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:00.948559 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:00.948596 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:00.950792 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:00.951306 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:00.951313 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:00.951315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:00.951417 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:00.951715 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:00.951719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:00.951748 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:00.952213 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:00.952435 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:00.952684 (+ 249us) server_negotiation.cc:299] Negotiation successful
0810 01:47:00.952927 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":144,"threads_started":1}
Aug 10 01:47:00 dist-test-slave-kvfs krb5kdc[14928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:47:00.966604 15065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:00.956997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37490 (local address 127.9.77.190:41633)
0810 01:47:00.957185 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:00.957191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:00.957210 (+ 19us) server_negotiation.cc:407] Connection header received
0810 01:47:00.957269 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:00.957271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:00.957322 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:00.957431 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:00.958149 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:00.958853 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:00.959774 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:00.960006 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:00.960210 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:00.962926 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:00.962930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:00.962932 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:00.962965 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:00.964597 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:00.965054 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:00.965058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:00.965059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:00.965154 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:00.965555 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:00.965558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:00.965560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:00.965860 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:00.965994 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:00.966305 (+ 311us) server_negotiation.cc:299] Negotiation successful
0810 01:47:00.966448 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260810 01:47:00.967198 14996 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37490
I20260810 01:47:00.968180 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 15103
I20260810 01:47:00.992143 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 15314
I20260810 01:47:01.006217 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 15531
I20260810 01:47:01.021237 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 14948
2026-08-10T01:47:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2666 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:47:01 dist-test-slave-kvfs krb5kdc[15750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:47:01 dist-test-slave-kvfs krb5kdc[15750](info): set up 2 sockets
Aug 10 01:47:01 dist-test-slave-kvfs krb5kdc[15750](info): commencing operation
krb5kdc: starting...
W20260810 01:47:02.395414 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.341s user 0.001s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:47:02 dist-test-slave-kvfs krb5kdc[15750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:47:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:47:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:47:02.540234 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:43395
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:43525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:43395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:02.678411 15772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:02.678826 15772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:02.678920 15772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:02.683293 15772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:47:02.683415 15772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:02.683454 15772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:02.683496 15772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:47:02.683530 15772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:47:02.689554 15772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:43525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:43395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:43395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:02.691074 15772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:02.692489 15772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:02.703111 15780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:02.703087 15782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:02.703100 15779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:02.703181 15772 server_base.cc:1034] running on GCE node
I20260810 01:47:02.704079 15772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:02.705595 15772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:02.706804 15772 hybrid_clock.cc:648] HybridClock initialized: now 1786326422706773 us; error 53 us; skew 500 ppm
Aug 10 01:47:02 dist-test-slave-kvfs krb5kdc[15750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326422, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:02.710290 15772 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:47:02.711205 15772 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:02.711315 15772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:02.711582 15772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:02.714217 15772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "0b4ba6db7ce24334a8f8e5e4493e20cf"
format_stamp: "Formatted at 2026-08-10 01:47:02 on dist-test-slave-kvfs"
server_key: "e0c69650b73c646d96bb963ccc1c7ce3"
server_key_iv: "831980a00a9c213a7dcc445a68ce9798"
server_key_version: "encryptionkey@0"
I20260810 01:47:02.714905 15772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "0b4ba6db7ce24334a8f8e5e4493e20cf"
format_stamp: "Formatted at 2026-08-10 01:47:02 on dist-test-slave-kvfs"
server_key: "e0c69650b73c646d96bb963ccc1c7ce3"
server_key_iv: "831980a00a9c213a7dcc445a68ce9798"
server_key_version: "encryptionkey@0"
I20260810 01:47:02.719651 15772 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 01:47:02.722939 15792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:02.724607 15772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 01:47:02.724788 15772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "0b4ba6db7ce24334a8f8e5e4493e20cf"
format_stamp: "Formatted at 2026-08-10 01:47:02 on dist-test-slave-kvfs"
server_key: "e0c69650b73c646d96bb963ccc1c7ce3"
server_key_iv: "831980a00a9c213a7dcc445a68ce9798"
server_key_version: "encryptionkey@0"
I20260810 01:47:02.724939 15772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:02.746424 15772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:02.750948 15772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:02.751231 15772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:02.760195 15772 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:43395
I20260810 01:47:02.760587 15869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:43395 every 8 connection(s)
I20260810 01:47:02.761137 15772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:47:02.764843 15870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:02.770082 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 15772
I20260810 01:47:02.770262 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:47:02.770614 9526 external_mini_cluster.cc:1428] Setting key caecbc7a9d164e47bc91bc16e63656c9
I20260810 01:47:02.772578 15870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b4ba6db7ce24334a8f8e5e4493e20cf: Bootstrap starting.
I20260810 01:47:02.776289 15870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b4ba6db7ce24334a8f8e5e4493e20cf: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:02.777462 15870 log.cc:824] T 00000000000000000000000000000000 P 0b4ba6db7ce24334a8f8e5e4493e20cf: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:02.780366 15870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b4ba6db7ce24334a8f8e5e4493e20cf: No bootstrap required, opened a new log
Aug 10 01:47:02 dist-test-slave-kvfs krb5kdc[15750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:47:02.785764 15870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b4ba6db7ce24334a8f8e5e4493e20cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4ba6db7ce24334a8f8e5e4493e20cf" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 43395 } }
I20260810 01:47:02.786128 15870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b4ba6db7ce24334a8f8e5e4493e20cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:02.786229 15870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b4ba6db7ce24334a8f8e5e4493e20cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b4ba6db7ce24334a8f8e5e4493e20cf, State: Initialized, Role: FOLLOWER
I20260810 01:47:02.786887 15870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b4ba6db7ce24334a8f8e5e4493e20cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4ba6db7ce24334a8f8e5e4493e20cf" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 43395 } }
I20260810 01:47:02.787117 15870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b4ba6db7ce24334a8f8e5e4493e20cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:47:02.787199 15870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b4ba6db7ce24334a8f8e5e4493e20cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:47:02.787314 15870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b4ba6db7ce24334a8f8e5e4493e20cf [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:02.788889 15870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b4ba6db7ce24334a8f8e5e4493e20cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4ba6db7ce24334a8f8e5e4493e20cf" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 43395 } }
I20260810 01:47:02.789342 15870 leader_election.cc:302] T 00000000000000000000000000000000 P 0b4ba6db7ce24334a8f8e5e4493e20cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b4ba6db7ce24334a8f8e5e4493e20cf; no voters:
I20260810 01:47:02.789767 15870 leader_election.cc:288] T 00000000000000000000000000000000 P 0b4ba6db7ce24334a8f8e5e4493e20cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:47:02.789929 15875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b4ba6db7ce24334a8f8e5e4493e20cf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:47:02.789947 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:02.772583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47018 (local address 127.9.77.190:43395)
0810 01:47:02.773156 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:02.773180 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:02.773245 (+ 65us) server_negotiation.cc:407] Connection header received
0810 01:47:02.774347 (+ 1102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:02.774381 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:02.774907 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:02.775379 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:02.776308 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:02.778014 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:02.779093 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:02.779855 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:02.780062 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:02.783314 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:02.783323 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:02.783347 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:02.783388 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:02.785817 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:02.786466 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:02.786474 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:02.786480 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:02.786643 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:02.786975 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:02.786978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:02.786980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:02.787691 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:02.787932 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:02.788266 (+ 334us) server_negotiation.cc:299] Negotiation successful
0810 01:47:02.788659 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":145,"threads_started":1}
I20260810 01:47:02.790170 15875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b4ba6db7ce24334a8f8e5e4493e20cf [term 1 LEADER]: Becoming Leader. State: Replica: 0b4ba6db7ce24334a8f8e5e4493e20cf, State: Running, Role: LEADER
I20260810 01:47:02.790542 15875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b4ba6db7ce24334a8f8e5e4493e20cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4ba6db7ce24334a8f8e5e4493e20cf" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 43395 } }
I20260810 01:47:02.791374 15870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b4ba6db7ce24334a8f8e5e4493e20cf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:47:02.793753 15876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b4ba6db7ce24334a8f8e5e4493e20cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b4ba6db7ce24334a8f8e5e4493e20cf. Latest consensus state: current_term: 1 leader_uuid: "0b4ba6db7ce24334a8f8e5e4493e20cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4ba6db7ce24334a8f8e5e4493e20cf" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 43395 } } }
I20260810 01:47:02.793896 15876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b4ba6db7ce24334a8f8e5e4493e20cf [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:02.794201 15877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b4ba6db7ce24334a8f8e5e4493e20cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b4ba6db7ce24334a8f8e5e4493e20cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4ba6db7ce24334a8f8e5e4493e20cf" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 43395 } } }
I20260810 01:47:02.794539 15877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b4ba6db7ce24334a8f8e5e4493e20cf [sys.catalog]: This master's current role is: LEADER
W20260810 01:47:02.797395 15894 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b4ba6db7ce24334a8f8e5e4493e20cf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 01:47:02.797494 15894 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 01:47:02.797601 15897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:47:02.798484 15897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:47:02.805781 15897 catalog_manager.cc:1347] Generated new cluster ID: ec6459ed153b4d68b8f2cc0a7d7d4d94
I20260810 01:47:02.805871 15897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:47:02.827586 15897 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:47:02.829293 15897 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:47:02.843252 15897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b4ba6db7ce24334a8f8e5e4493e20cf: Generated new TSK 0
I20260810 01:47:02.844287 15897 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.77.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
I20260810 01:47:02.892249 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:0
--local_ip_for_outbound_sockets=127.9.77.129
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:43395
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:43525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:03.039610 15902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:03.039945 15902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:03.039994 15902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:03.044234 15902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:03.044310 15902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:03.044404 15902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:47:03.050048 15902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:43525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:43395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:03.051478 15902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:03.052811 15902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:03.064208 15908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:03.064213 15909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:03.064213 15911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:03.064631 15902 server_base.cc:1034] running on GCE node
I20260810 01:47:03.065392 15902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:03.066164 15902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:03.067358 15902 hybrid_clock.cc:648] HybridClock initialized: now 1786326423067321 us; error 50 us; skew 500 ppm
Aug 10 01:47:03 dist-test-slave-kvfs krb5kdc[15750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326423, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:03.071066 15902 init.cc:377] Logged in from keytab as kudu/127.9.77.129@KRBTEST.COM (short username kudu)
I20260810 01:47:03.072137 15902 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:03.072216 15902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:03.072510 15902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:03.075182 15902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-0/data/instance:
uuid: "0ac36ea9a0f54d369864a4b4789a1570"
format_stamp: "Formatted at 2026-08-10 01:47:03 on dist-test-slave-kvfs"
server_key: "ee8fd8a558de677c5bccc0e8e184c86d"
server_key_iv: "7437332450376423b68633ffd58a1c23"
server_key_version: "encryptionkey@0"
I20260810 01:47:03.075887 15902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance:
uuid: "0ac36ea9a0f54d369864a4b4789a1570"
format_stamp: "Formatted at 2026-08-10 01:47:03 on dist-test-slave-kvfs"
server_key: "ee8fd8a558de677c5bccc0e8e184c86d"
server_key_iv: "7437332450376423b68633ffd58a1c23"
server_key_version: "encryptionkey@0"
I20260810 01:47:03.080431 15902 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 01:47:03.083305 15922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:03.084903 15902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:47:03.085073 15902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "0ac36ea9a0f54d369864a4b4789a1570"
format_stamp: "Formatted at 2026-08-10 01:47:03 on dist-test-slave-kvfs"
server_key: "ee8fd8a558de677c5bccc0e8e184c86d"
server_key_iv: "7437332450376423b68633ffd58a1c23"
server_key_version: "encryptionkey@0"
I20260810 01:47:03.085192 15902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:03.107141 15902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:03.110873 15902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:03.111114 15902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:03.111857 15902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:03.113045 15902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:03.113097 15902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:03.113173 15902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:03.113204 15902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:03.123996 15902 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:42841
I20260810 01:47:03.123996 16093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:42841 every 8 connection(s)
I20260810 01:47:03.124827 15902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:47:03.134049 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 15902
I20260810 01:47:03.134232 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance
I20260810 01:47:03.134644 9526 external_mini_cluster.cc:1428] Setting key c4a5f28f72f44d5671e6eac2cbaee247
Aug 10 01:47:03 dist-test-slave-kvfs krb5kdc[15750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326423, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:47:03.142796 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:03.127207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:51821 (local address 127.9.77.190:43395)
0810 01:47:03.127379 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:03.127385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:03.128956 (+ 1571us) server_negotiation.cc:407] Connection header received
0810 01:47:03.130070 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:03.130073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:03.130135 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:03.130226 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:03.131845 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.132555 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.134275 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.134656 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.134746 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:03.138220 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:03.138223 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:03.138227 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:03.138265 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:03.140310 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:03.140882 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:03.140887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:03.140889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:03.141000 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:03.141358 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:03.141363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:03.141367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:03.141628 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:03.141714 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:03.142333 (+ 619us) server_negotiation.cc:299] Negotiation successful
0810 01:47:03.142515 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 01:47:03.143783 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:03.127683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:43395 (local address 127.9.77.129:51821)
0810 01:47:03.128756 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:03.128807 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:03.129831 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:03.130536 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:03.130544 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:03.131010 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:03.131632 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:03.131648 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.132688 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.132692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:03.134114 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:03.134127 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.135013 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.135019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:03.135246 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:03.136085 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:03.136111 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:03.138022 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:03.140454 (+ 2432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:03.140460 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:03.140478 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:03.140771 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:03.141102 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:03.141104 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:03.141105 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:03.141253 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:03.141723 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:03.141729 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:03.142054 (+ 325us) client_negotiation.cc:769] Sending connection context
0810 01:47:03.142242 (+ 188us) client_negotiation.cc:240] Negotiation successful
0810 01:47:03.142589 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":146,"threads_started":1}
I20260810 01:47:03.145433 16094 heartbeater.cc:347] Connected to a master server at 127.9.77.190:43395
I20260810 01:47:03.145728 16094 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:03.146325 16094 heartbeater.cc:511] Master 127.9.77.190:43395 requested a full tablet report, sending...
I20260810 01:47:03.148345 15816 ts_manager.cc:194] Registered new tserver with Master: 0ac36ea9a0f54d369864a4b4789a1570 (127.9.77.129:42841)
WARNING: no policy specified for kudu/127.9.77.130@KRBTEST.COM; defaulting to no policy
I20260810 01:47:03.149803 15816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.129@KRBTEST.COM'} at 127.9.77.129:51821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
I20260810 01:47:03.172528 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:0
--local_ip_for_outbound_sockets=127.9.77.130
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:43395
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:43525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:03.310392 16101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:03.310768 16101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:03.310859 16101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:03.315236 16101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:03.315340 16101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:03.315439 16101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:47:03.321105 16101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:43525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:43395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:03.322546 16101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:03.323930 16101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:03.335579 16110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:03.335574 16108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:03.335574 16107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:03.336136 16101 server_base.cc:1034] running on GCE node
I20260810 01:47:03.336596 16101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:03.337344 16101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:03.338531 16101 hybrid_clock.cc:648] HybridClock initialized: now 1786326423338492 us; error 50 us; skew 500 ppm
Aug 10 01:47:03 dist-test-slave-kvfs krb5kdc[15750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326423, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:03.341933 16101 init.cc:377] Logged in from keytab as kudu/127.9.77.130@KRBTEST.COM (short username kudu)
I20260810 01:47:03.342907 16101 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:03.343014 16101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:03.343297 16101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:03.345837 16101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-1/data/instance:
uuid: "e1d7065448cd4ec0935b49302556cda5"
format_stamp: "Formatted at 2026-08-10 01:47:03 on dist-test-slave-kvfs"
server_key: "e852695f67521378ad827ee498539785"
server_key_iv: "c88334d8177f017958521a54df2695b2"
server_key_version: "encryptionkey@0"
I20260810 01:47:03.346472 16101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance:
uuid: "e1d7065448cd4ec0935b49302556cda5"
format_stamp: "Formatted at 2026-08-10 01:47:03 on dist-test-slave-kvfs"
server_key: "e852695f67521378ad827ee498539785"
server_key_iv: "c88334d8177f017958521a54df2695b2"
server_key_version: "encryptionkey@0"
I20260810 01:47:03.350978 16101 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 01:47:03.353986 16117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:03.355512 16101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 01:47:03.355665 16101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "e1d7065448cd4ec0935b49302556cda5"
format_stamp: "Formatted at 2026-08-10 01:47:03 on dist-test-slave-kvfs"
server_key: "e852695f67521378ad827ee498539785"
server_key_iv: "c88334d8177f017958521a54df2695b2"
server_key_version: "encryptionkey@0"
I20260810 01:47:03.355795 16101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:03.366176 16101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:03.369753 16101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:03.370011 16101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:03.370726 16101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:03.371747 16101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:03.371824 16101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:03.371903 16101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:03.371949 16101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:03.383044 16101 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:37683
I20260810 01:47:03.383064 16283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:37683 every 8 connection(s)
I20260810 01:47:03.383873 16101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:47:03.392380 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 16101
I20260810 01:47:03.392542 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance
I20260810 01:47:03.392958 9526 external_mini_cluster.cc:1428] Setting key c27843754d78395287a854ceb279bdaf
Aug 10 01:47:03 dist-test-slave-kvfs krb5kdc[15750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326423, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:47:03.402134 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:03.386280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:43233 (local address 127.9.77.190:43395)
0810 01:47:03.386498 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:03.386504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:03.387592 (+ 1088us) server_negotiation.cc:407] Connection header received
0810 01:47:03.389487 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:03.389490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:03.389556 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:03.389647 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:03.391285 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.392006 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.393823 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.394135 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.394235 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:03.397538 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:03.397546 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:03.397552 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:03.397597 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:03.399482 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:03.400192 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:03.400196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:03.400198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:03.400302 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:03.400700 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:03.400703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:03.400705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:03.400965 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:03.401086 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:03.401737 (+ 651us) server_negotiation.cc:299] Negotiation successful
0810 01:47:03.401925 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260810 01:47:03.403404 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:03.386791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:43395 (local address 127.9.77.130:43233)
0810 01:47:03.387388 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:03.387442 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:03.389167 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:03.389861 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:03.389892 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:03.390428 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:03.391108 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:03.391124 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.392218 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.392224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:03.393699 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:03.393710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.394314 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.394318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:03.394544 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:03.395319 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:03.395339 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:03.397326 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:03.399648 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:03.399656 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:03.399688 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:03.400067 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:03.400407 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:03.400411 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:03.400413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:03.400591 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:03.401083 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:03.401092 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:03.401449 (+ 357us) client_negotiation.cc:769] Sending connection context
0810 01:47:03.401681 (+ 232us) client_negotiation.cc:240] Negotiation successful
0810 01:47:03.402130 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":138,"threads_started":1}
I20260810 01:47:03.405249 16284 heartbeater.cc:347] Connected to a master server at 127.9.77.190:43395
I20260810 01:47:03.405591 16284 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:03.406284 16284 heartbeater.cc:511] Master 127.9.77.190:43395 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.77.131@KRBTEST.COM; defaulting to no policy
I20260810 01:47:03.407788 15816 ts_manager.cc:194] Registered new tserver with Master: e1d7065448cd4ec0935b49302556cda5 (127.9.77.130:37683)
I20260810 01:47:03.408548 15816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.130@KRBTEST.COM'} at 127.9.77.130:43233
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
I20260810 01:47:03.428531 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:0
--local_ip_for_outbound_sockets=127.9.77.131
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:43395
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:43525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:03.564002 16289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:03.564376 16289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:03.564432 16289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:03.568684 16289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:03.568764 16289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:03.568831 16289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:47:03.574553 16289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:43525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:43395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:03.575932 16289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:03.577359 16289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:03.588938 16298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:03.588938 16300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:03.588984 16289 server_base.cc:1034] running on GCE node
W20260810 01:47:03.589187 16302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:03.589679 16289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:03.590472 16289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:03.591631 16289 hybrid_clock.cc:648] HybridClock initialized: now 1786326423591602 us; error 45 us; skew 500 ppm
Aug 10 01:47:03 dist-test-slave-kvfs krb5kdc[15750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326423, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:03.595156 16289 init.cc:377] Logged in from keytab as kudu/127.9.77.131@KRBTEST.COM (short username kudu)
I20260810 01:47:03.596160 16289 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:03.596240 16289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:03.596537 16289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:03.599120 16289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-2/data/instance:
uuid: "9d953e27874844fb848a8239b3671f80"
format_stamp: "Formatted at 2026-08-10 01:47:03 on dist-test-slave-kvfs"
server_key: "fdcc4711c6c28acaa59a3ec693d0624f"
server_key_iv: "ea17eb1b40d79c0eecff5574861657d5"
server_key_version: "encryptionkey@0"
I20260810 01:47:03.600219 16289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance:
uuid: "9d953e27874844fb848a8239b3671f80"
format_stamp: "Formatted at 2026-08-10 01:47:03 on dist-test-slave-kvfs"
server_key: "fdcc4711c6c28acaa59a3ec693d0624f"
server_key_iv: "ea17eb1b40d79c0eecff5574861657d5"
server_key_version: "encryptionkey@0"
I20260810 01:47:03.605032 16289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 01:47:03.608251 16311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:03.609817 16289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 01:47:03.609961 16289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "9d953e27874844fb848a8239b3671f80"
format_stamp: "Formatted at 2026-08-10 01:47:03 on dist-test-slave-kvfs"
server_key: "fdcc4711c6c28acaa59a3ec693d0624f"
server_key_iv: "ea17eb1b40d79c0eecff5574861657d5"
server_key_version: "encryptionkey@0"
I20260810 01:47:03.610095 16289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:03.636734 16289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:03.640661 16289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:03.640903 16289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:03.641674 16289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:03.642736 16289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:03.642789 16289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:03.642838 16289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:03.642891 16289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:03.653573 16289 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:34691
I20260810 01:47:03.653589 16483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:34691 every 8 connection(s)
I20260810 01:47:03.654366 16289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:47:03.658361 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 16289
I20260810 01:47:03.658514 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance
I20260810 01:47:03.658953 9526 external_mini_cluster.cc:1428] Setting key d7e66d3bece8a0e08fb014ecb9fa4865
Aug 10 01:47:03 dist-test-slave-kvfs krb5kdc[15750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326423, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:47:03.670410 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:03.656681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:37007 (local address 127.9.77.190:43395)
0810 01:47:03.656881 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:03.656885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:03.658396 (+ 1511us) server_negotiation.cc:407] Connection header received
0810 01:47:03.659636 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:03.659641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:03.659718 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:03.659870 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:03.661552 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.662213 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.663618 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.663862 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.663946 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:03.666813 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:03.666816 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:03.666818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:03.666854 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:03.668287 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:03.668794 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:03.668797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:03.668798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:03.668896 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:03.669247 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:03.669249 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:03.669250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:03.669445 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:03.669525 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:03.670110 (+ 585us) server_negotiation.cc:299] Negotiation successful
0810 01:47:03.670243 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 01:47:03.671386 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:03.657150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:43395 (local address 127.9.77.131:37007)
0810 01:47:03.658203 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:03.658257 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:03.659376 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:03.660128 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:03.660137 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:03.660590 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:03.661378 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:03.661399 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.662496 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.662499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:03.663498 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:03.663508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.664071 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.664079 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:03.664262 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:03.664958 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:03.664977 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:03.666637 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:03.668394 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:03.668400 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:03.668417 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:03.668696 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:03.669001 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:03.669004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:03.669005 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:03.669165 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:03.669535 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:03.669540 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:03.669819 (+ 279us) client_negotiation.cc:769] Sending connection context
0810 01:47:03.670020 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 01:47:03.670333 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":144,"threads_started":1}
I20260810 01:47:03.672796 16484 heartbeater.cc:347] Connected to a master server at 127.9.77.190:43395
I20260810 01:47:03.673130 16484 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:03.673763 16484 heartbeater.cc:511] Master 127.9.77.190:43395 requested a full tablet report, sending...
I20260810 01:47:03.674938 15816 ts_manager.cc:194] Registered new tserver with Master: 9d953e27874844fb848a8239b3671f80 (127.9.77.131:34691)
I20260810 01:47:03.675591 15816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.131@KRBTEST.COM'} at 127.9.77.131:37007
I20260810 01:47:03.685603 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 01:47:03.697342 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:03.688042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47020 (local address 127.9.77.190:43395)
0810 01:47:03.688205 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:03.688210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:03.688381 (+ 171us) server_negotiation.cc:407] Connection header received
0810 01:47:03.688590 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:03.688593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:03.688643 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:03.688737 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:03.689708 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.690427 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.691397 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.691717 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.691838 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:03.693599 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:03.693603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:03.693607 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:03.693640 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:03.695205 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:03.695729 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:03.695736 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:03.695740 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:03.695838 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:03.696236 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:03.696240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:03.696242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:03.696526 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:03.696690 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:03.696983 (+ 293us) server_negotiation.cc:299] Negotiation successful
0810 01:47:03.697120 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 01:47:03.701193 15816 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 01:47:03.704973 15816 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:47:03.722731 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:03.716527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:42841 (local address 127.0.0.1:51732)
0810 01:47:03.717143 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:03.717160 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:03.717248 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:03.718560 (+ 1312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:03.718564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:03.718585 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:03.718688 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:03.718698 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.720092 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.720095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:03.720905 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:03.720914 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.722024 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.722027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:03.722137 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:03.722179 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 01:47:03.722300 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 01:47:03.722551 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":2,"thread_start_us":115,"threads_started":1}
I20260810 01:47:03.722738 16502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:03.717066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51732 (local address 127.9.77.129:42841)
0810 01:47:03.718060 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:03.718066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:03.718106 (+ 40us) server_negotiation.cc:407] Connection header received
0810 01:47:03.718191 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:03.718196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:03.718393 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:03.718550 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:03.718822 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.719945 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.721034 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.721873 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.721995 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:03.722178 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:03.722417 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 01:47:03.722541 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":871,"thread_start_us":117,"threads_started":1}
I20260810 01:47:03.723086 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:03.715636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:34691 (local address 127.0.0.1:36696)
0810 01:47:03.716648 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:03.716706 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:03.716873 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:03.717780 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:03.717787 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:03.717822 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:03.718094 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:03.718102 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.719473 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.719476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:03.721185 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:03.721197 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.722258 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.722273 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:03.722507 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:03.722532 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 01:47:03.722614 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 01:47:03.722675 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":273,"spinlock_wait_cycles":4480,"thread_start_us":104,"threads_started":1}
I20260810 01:47:03.723423 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:03.716439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48096 (local address 127.9.77.130:37683)
0810 01:47:03.717263 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:03.717278 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:03.717305 (+ 27us) server_negotiation.cc:407] Connection header received
0810 01:47:03.717411 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:03.717415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:03.717612 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:03.717791 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:03.718169 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.719409 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.721322 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.722152 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.722257 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:03.722560 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:03.723153 (+ 593us) server_negotiation.cc:299] Negotiation successful
0810 01:47:03.723258 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":145,"threads_started":1}
I20260810 01:47:03.723662 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:03.715847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:37683 (local address 127.0.0.1:48096)
0810 01:47:03.717106 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:03.717121 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:03.717222 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:03.717780 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:03.717787 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:03.717831 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:03.718021 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:03.718029 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.719583 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.719587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:03.721185 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:03.721197 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.722260 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.722264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:03.722832 (+ 568us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:03.722863 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 01:47:03.722940 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 01:47:03.723017 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1177,"mutex_wait_us":585,"spinlock_wait_cycles":2304,"thread_start_us":120,"threads_started":1}
I20260810 01:47:03.723858 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:03.715785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36696 (local address 127.9.77.131:34691)
0810 01:47:03.716234 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:03.716248 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:03.716801 (+ 553us) server_negotiation.cc:407] Connection header received
0810 01:47:03.717453 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:03.717458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:03.717614 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:03.717791 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:03.718228 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.719314 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.721346 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.721906 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.722039 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:03.722194 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:03.723556 (+ 1362us) server_negotiation.cc:299] Negotiation successful
0810 01:47:03.723672 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":149,"threads_started":1}
I20260810 01:47:03.726892 15990 tablet_service.cc:1467] Processing CreateTablet for tablet 0f43293f9a4f463d84ef1641d53fc3c4 (DEFAULT_TABLE table=test-table [id=4c3fc8396f664b4d889e871fe1636609]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:47:03.726888 16188 tablet_service.cc:1467] Processing CreateTablet for tablet 0f43293f9a4f463d84ef1641d53fc3c4 (DEFAULT_TABLE table=test-table [id=4c3fc8396f664b4d889e871fe1636609]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:47:03.728396 15990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f43293f9a4f463d84ef1641d53fc3c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:03.728628 16188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f43293f9a4f463d84ef1641d53fc3c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:03.730258 16379 tablet_service.cc:1467] Processing CreateTablet for tablet 0f43293f9a4f463d84ef1641d53fc3c4 (DEFAULT_TABLE table=test-table [id=4c3fc8396f664b4d889e871fe1636609]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:47:03.731343 16379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f43293f9a4f463d84ef1641d53fc3c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:03.737258 16505 tablet_bootstrap.cc:492] T 0f43293f9a4f463d84ef1641d53fc3c4 P e1d7065448cd4ec0935b49302556cda5: Bootstrap starting.
I20260810 01:47:03.738044 16504 tablet_bootstrap.cc:492] T 0f43293f9a4f463d84ef1641d53fc3c4 P 0ac36ea9a0f54d369864a4b4789a1570: Bootstrap starting.
I20260810 01:47:03.738708 16509 tablet_bootstrap.cc:492] T 0f43293f9a4f463d84ef1641d53fc3c4 P 9d953e27874844fb848a8239b3671f80: Bootstrap starting.
I20260810 01:47:03.740983 16509 tablet_bootstrap.cc:654] T 0f43293f9a4f463d84ef1641d53fc3c4 P 9d953e27874844fb848a8239b3671f80: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:03.741098 16505 tablet_bootstrap.cc:654] T 0f43293f9a4f463d84ef1641d53fc3c4 P e1d7065448cd4ec0935b49302556cda5: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:03.741348 16504 tablet_bootstrap.cc:654] T 0f43293f9a4f463d84ef1641d53fc3c4 P 0ac36ea9a0f54d369864a4b4789a1570: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:03.742269 16509 log.cc:824] T 0f43293f9a4f463d84ef1641d53fc3c4 P 9d953e27874844fb848a8239b3671f80: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:03.742301 16505 log.cc:824] T 0f43293f9a4f463d84ef1641d53fc3c4 P e1d7065448cd4ec0935b49302556cda5: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:03.742312 16504 log.cc:824] T 0f43293f9a4f463d84ef1641d53fc3c4 P 0ac36ea9a0f54d369864a4b4789a1570: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:03.744436 16509 tablet_bootstrap.cc:492] T 0f43293f9a4f463d84ef1641d53fc3c4 P 9d953e27874844fb848a8239b3671f80: No bootstrap required, opened a new log
I20260810 01:47:03.744432 16505 tablet_bootstrap.cc:492] T 0f43293f9a4f463d84ef1641d53fc3c4 P e1d7065448cd4ec0935b49302556cda5: No bootstrap required, opened a new log
I20260810 01:47:03.744432 16504 tablet_bootstrap.cc:492] T 0f43293f9a4f463d84ef1641d53fc3c4 P 0ac36ea9a0f54d369864a4b4789a1570: No bootstrap required, opened a new log
I20260810 01:47:03.744661 16509 ts_tablet_manager.cc:1397] T 0f43293f9a4f463d84ef1641d53fc3c4 P 9d953e27874844fb848a8239b3671f80: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 01:47:03.744663 16505 ts_tablet_manager.cc:1397] T 0f43293f9a4f463d84ef1641d53fc3c4 P e1d7065448cd4ec0935b49302556cda5: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 01:47:03.744663 16504 ts_tablet_manager.cc:1397] T 0f43293f9a4f463d84ef1641d53fc3c4 P 0ac36ea9a0f54d369864a4b4789a1570: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 01:47:03.748229 16509 raft_consensus.cc:348] T 0f43293f9a4f463d84ef1641d53fc3c4 P 9d953e27874844fb848a8239b3671f80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d953e27874844fb848a8239b3671f80" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 34691 } } peers { permanent_uuid: "e1d7065448cd4ec0935b49302556cda5" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37683 } } peers { permanent_uuid: "0ac36ea9a0f54d369864a4b4789a1570" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 42841 } }
I20260810 01:47:03.748544 16509 raft_consensus.cc:374] T 0f43293f9a4f463d84ef1641d53fc3c4 P 9d953e27874844fb848a8239b3671f80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:03.748601 16509 raft_consensus.cc:729] T 0f43293f9a4f463d84ef1641d53fc3c4 P 9d953e27874844fb848a8239b3671f80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d953e27874844fb848a8239b3671f80, State: Initialized, Role: FOLLOWER
I20260810 01:47:03.748970 16505 raft_consensus.cc:348] T 0f43293f9a4f463d84ef1641d53fc3c4 P e1d7065448cd4ec0935b49302556cda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d953e27874844fb848a8239b3671f80" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 34691 } } peers { permanent_uuid: "e1d7065448cd4ec0935b49302556cda5" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37683 } } peers { permanent_uuid: "0ac36ea9a0f54d369864a4b4789a1570" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 42841 } }
I20260810 01:47:03.749116 16509 consensus_queue.cc:260] T 0f43293f9a4f463d84ef1641d53fc3c4 P 9d953e27874844fb848a8239b3671f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d953e27874844fb848a8239b3671f80" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 34691 } } peers { permanent_uuid: "e1d7065448cd4ec0935b49302556cda5" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37683 } } peers { permanent_uuid: "0ac36ea9a0f54d369864a4b4789a1570" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 42841 } }
I20260810 01:47:03.749249 16505 raft_consensus.cc:374] T 0f43293f9a4f463d84ef1641d53fc3c4 P e1d7065448cd4ec0935b49302556cda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:03.749317 16505 raft_consensus.cc:729] T 0f43293f9a4f463d84ef1641d53fc3c4 P e1d7065448cd4ec0935b49302556cda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1d7065448cd4ec0935b49302556cda5, State: Initialized, Role: FOLLOWER
I20260810 01:47:03.749485 16504 raft_consensus.cc:348] T 0f43293f9a4f463d84ef1641d53fc3c4 P 0ac36ea9a0f54d369864a4b4789a1570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d953e27874844fb848a8239b3671f80" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 34691 } } peers { permanent_uuid: "e1d7065448cd4ec0935b49302556cda5" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37683 } } peers { permanent_uuid: "0ac36ea9a0f54d369864a4b4789a1570" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 42841 } }
I20260810 01:47:03.749725 16504 raft_consensus.cc:374] T 0f43293f9a4f463d84ef1641d53fc3c4 P 0ac36ea9a0f54d369864a4b4789a1570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:03.749782 16504 raft_consensus.cc:729] T 0f43293f9a4f463d84ef1641d53fc3c4 P 0ac36ea9a0f54d369864a4b4789a1570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac36ea9a0f54d369864a4b4789a1570, State: Initialized, Role: FOLLOWER
I20260810 01:47:03.749768 16505 consensus_queue.cc:260] T 0f43293f9a4f463d84ef1641d53fc3c4 P e1d7065448cd4ec0935b49302556cda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d953e27874844fb848a8239b3671f80" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 34691 } } peers { permanent_uuid: "e1d7065448cd4ec0935b49302556cda5" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37683 } } peers { permanent_uuid: "0ac36ea9a0f54d369864a4b4789a1570" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 42841 } }
I20260810 01:47:03.750193 16504 consensus_queue.cc:260] T 0f43293f9a4f463d84ef1641d53fc3c4 P 0ac36ea9a0f54d369864a4b4789a1570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d953e27874844fb848a8239b3671f80" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 34691 } } peers { permanent_uuid: "e1d7065448cd4ec0935b49302556cda5" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37683 } } peers { permanent_uuid: "0ac36ea9a0f54d369864a4b4789a1570" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 42841 } }
I20260810 01:47:03.750428 16509 ts_tablet_manager.cc:1428] T 0f43293f9a4f463d84ef1641d53fc3c4 P 9d953e27874844fb848a8239b3671f80: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 01:47:03.750593 16284 heartbeater.cc:503] Master 127.9.77.190:43395 was elected leader, sending a full tablet report...
I20260810 01:47:03.750799 16484 heartbeater.cc:503] Master 127.9.77.190:43395 was elected leader, sending a full tablet report...
I20260810 01:47:03.751178 16504 ts_tablet_manager.cc:1428] T 0f43293f9a4f463d84ef1641d53fc3c4 P 0ac36ea9a0f54d369864a4b4789a1570: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 01:47:03.751544 16094 heartbeater.cc:503] Master 127.9.77.190:43395 was elected leader, sending a full tablet report...
I20260810 01:47:03.752161 16505 ts_tablet_manager.cc:1428] T 0f43293f9a4f463d84ef1641d53fc3c4 P e1d7065448cd4ec0935b49302556cda5: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260810 01:47:03.876519 16096 tablet.cc:2369] T 0f43293f9a4f463d84ef1641d53fc3c4 P 0ac36ea9a0f54d369864a4b4789a1570: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 01:47:03.885298 16285 tablet.cc:2369] T 0f43293f9a4f463d84ef1641d53fc3c4 P e1d7065448cd4ec0935b49302556cda5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 01:47:03.905565 16485 tablet.cc:2369] T 0f43293f9a4f463d84ef1641d53fc3c4 P 9d953e27874844fb848a8239b3671f80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 01:47:03.913885 16518 raft_consensus.cc:482] T 0f43293f9a4f463d84ef1641d53fc3c4 P e1d7065448cd4ec0935b49302556cda5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:47:03.914318 16518 raft_consensus.cc:504] T 0f43293f9a4f463d84ef1641d53fc3c4 P e1d7065448cd4ec0935b49302556cda5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d953e27874844fb848a8239b3671f80" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 34691 } } peers { permanent_uuid: "e1d7065448cd4ec0935b49302556cda5" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37683 } } peers { permanent_uuid: "0ac36ea9a0f54d369864a4b4789a1570" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 42841 } }
I20260810 01:47:03.916220 16518 leader_election.cc:288] T 0f43293f9a4f463d84ef1641d53fc3c4 P e1d7065448cd4ec0935b49302556cda5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d953e27874844fb848a8239b3671f80 (127.9.77.131:34691), 0ac36ea9a0f54d369864a4b4789a1570 (127.9.77.129:42841)
I20260810 01:47:03.921183 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:03.916589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:34691 (local address 127.9.77.130:34703)
0810 01:47:03.917152 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:03.917166 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:03.917326 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:03.917638 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:03.917642 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:03.917659 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:03.917871 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:03.917878 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.918736 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.918740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:03.919794 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:03.919805 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.920605 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.920609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:03.920718 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:03.920777 (+ 59us) client_negotiation.cc:769] Sending connection context
0810 01:47:03.920880 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 01:47:03.920971 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":67,"spinlock_wait_cycles":111488,"thread_start_us":207,"threads_started":1}
I20260810 01:47:03.921160 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:03.916727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:34703 (local address 127.9.77.131:34691)
0810 01:47:03.916959 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:03.916966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:03.917271 (+ 305us) server_negotiation.cc:407] Connection header received
0810 01:47:03.917451 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:03.917454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:03.917520 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:03.917622 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:03.917994 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.918604 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.919931 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.920461 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.920609 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:03.920685 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:03.920897 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 01:47:03.920991 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260810 01:47:03.921170 16502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:03.916886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:59579 (local address 127.9.77.129:42841)
0810 01:47:03.917057 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:03.917062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:03.917228 (+ 166us) server_negotiation.cc:407] Connection header received
0810 01:47:03.917445 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:03.917450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:03.917507 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:03.917610 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:03.918013 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.918574 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.919938 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.920492 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.920639 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:03.920724 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:03.920940 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 01:47:03.921024 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 01:47:03.921204 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:03.916570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:42841 (local address 127.9.77.130:59579)
0810 01:47:03.917086 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:03.917104 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:03.917306 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:03.917625 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:03.917629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:03.917647 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:03.917883 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:03.917890 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.918700 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.918705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:03.919785 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:03.919796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.920627 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.920631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:03.920786 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:03.920819 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 01:47:03.920922 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 01:47:03.921004 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":94,"spinlock_wait_cycles":3328,"thread_start_us":190,"threads_started":1}
I20260810 01:47:03.922178 16417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f43293f9a4f463d84ef1641d53fc3c4" candidate_uuid: "e1d7065448cd4ec0935b49302556cda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d953e27874844fb848a8239b3671f80" is_pre_election: true
I20260810 01:47:03.922111 16021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f43293f9a4f463d84ef1641d53fc3c4" candidate_uuid: "e1d7065448cd4ec0935b49302556cda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac36ea9a0f54d369864a4b4789a1570" is_pre_election: true
I20260810 01:47:03.922603 16417 raft_consensus.cc:2454] T 0f43293f9a4f463d84ef1641d53fc3c4 P 9d953e27874844fb848a8239b3671f80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1d7065448cd4ec0935b49302556cda5 in term 0.
I20260810 01:47:03.922603 16021 raft_consensus.cc:2454] T 0f43293f9a4f463d84ef1641d53fc3c4 P 0ac36ea9a0f54d369864a4b4789a1570 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1d7065448cd4ec0935b49302556cda5 in term 0.
I20260810 01:47:03.923317 16118 leader_election.cc:302] T 0f43293f9a4f463d84ef1641d53fc3c4 P e1d7065448cd4ec0935b49302556cda5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ac36ea9a0f54d369864a4b4789a1570, 9d953e27874844fb848a8239b3671f80, e1d7065448cd4ec0935b49302556cda5; no voters:
I20260810 01:47:03.923729 16518 raft_consensus.cc:2781] T 0f43293f9a4f463d84ef1641d53fc3c4 P e1d7065448cd4ec0935b49302556cda5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:47:03.923827 16518 raft_consensus.cc:482] T 0f43293f9a4f463d84ef1641d53fc3c4 P e1d7065448cd4ec0935b49302556cda5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:47:03.923893 16518 raft_consensus.cc:3037] T 0f43293f9a4f463d84ef1641d53fc3c4 P e1d7065448cd4ec0935b49302556cda5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:03.925374 16518 raft_consensus.cc:504] T 0f43293f9a4f463d84ef1641d53fc3c4 P e1d7065448cd4ec0935b49302556cda5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d953e27874844fb848a8239b3671f80" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 34691 } } peers { permanent_uuid: "e1d7065448cd4ec0935b49302556cda5" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37683 } } peers { permanent_uuid: "0ac36ea9a0f54d369864a4b4789a1570" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 42841 } }
I20260810 01:47:03.925884 16518 leader_election.cc:288] T 0f43293f9a4f463d84ef1641d53fc3c4 P e1d7065448cd4ec0935b49302556cda5 [CANDIDATE]: Term 1 election: Requested vote from peers 9d953e27874844fb848a8239b3671f80 (127.9.77.131:34691), 0ac36ea9a0f54d369864a4b4789a1570 (127.9.77.129:42841)
I20260810 01:47:03.926419 16417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f43293f9a4f463d84ef1641d53fc3c4" candidate_uuid: "e1d7065448cd4ec0935b49302556cda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d953e27874844fb848a8239b3671f80"
I20260810 01:47:03.926419 16021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f43293f9a4f463d84ef1641d53fc3c4" candidate_uuid: "e1d7065448cd4ec0935b49302556cda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac36ea9a0f54d369864a4b4789a1570"
I20260810 01:47:03.926589 16417 raft_consensus.cc:3037] T 0f43293f9a4f463d84ef1641d53fc3c4 P 9d953e27874844fb848a8239b3671f80 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:03.926590 16021 raft_consensus.cc:3037] T 0f43293f9a4f463d84ef1641d53fc3c4 P 0ac36ea9a0f54d369864a4b4789a1570 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:03.928279 16417 raft_consensus.cc:2454] T 0f43293f9a4f463d84ef1641d53fc3c4 P 9d953e27874844fb848a8239b3671f80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1d7065448cd4ec0935b49302556cda5 in term 1.
I20260810 01:47:03.928269 16021 raft_consensus.cc:2454] T 0f43293f9a4f463d84ef1641d53fc3c4 P 0ac36ea9a0f54d369864a4b4789a1570 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1d7065448cd4ec0935b49302556cda5 in term 1.
I20260810 01:47:03.928751 16122 leader_election.cc:302] T 0f43293f9a4f463d84ef1641d53fc3c4 P e1d7065448cd4ec0935b49302556cda5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ac36ea9a0f54d369864a4b4789a1570, e1d7065448cd4ec0935b49302556cda5; no voters:
I20260810 01:47:03.929026 16518 raft_consensus.cc:2781] T 0f43293f9a4f463d84ef1641d53fc3c4 P e1d7065448cd4ec0935b49302556cda5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:47:03.929368 16518 raft_consensus.cc:686] T 0f43293f9a4f463d84ef1641d53fc3c4 P e1d7065448cd4ec0935b49302556cda5 [term 1 LEADER]: Becoming Leader. State: Replica: e1d7065448cd4ec0935b49302556cda5, State: Running, Role: LEADER
I20260810 01:47:03.929845 16518 consensus_queue.cc:237] T 0f43293f9a4f463d84ef1641d53fc3c4 P e1d7065448cd4ec0935b49302556cda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d953e27874844fb848a8239b3671f80" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 34691 } } peers { permanent_uuid: "e1d7065448cd4ec0935b49302556cda5" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37683 } } peers { permanent_uuid: "0ac36ea9a0f54d369864a4b4789a1570" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 42841 } }
I20260810 01:47:03.934317 15812 catalog_manager.cc:5495] T 0f43293f9a4f463d84ef1641d53fc3c4 P e1d7065448cd4ec0935b49302556cda5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1d7065448cd4ec0935b49302556cda5 (127.9.77.130). New cstate: current_term: 1 leader_uuid: "e1d7065448cd4ec0935b49302556cda5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d953e27874844fb848a8239b3671f80" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 34691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1d7065448cd4ec0935b49302556cda5" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ac36ea9a0f54d369864a4b4789a1570" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 42841 } health_report { overall_health: UNKNOWN } } }
I20260810 01:47:03.963836 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:03.959651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48104 (local address 127.9.77.130:37683)
0810 01:47:03.959813 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:03.959818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:03.959836 (+ 18us) server_negotiation.cc:407] Connection header received
0810 01:47:03.959960 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:03.959963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:03.960056 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:03.960177 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:47:03.960460 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.961229 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.962126 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:03.962370 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:03.962470 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:03.962777 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:47:03.963356 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:47:03.963467 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:03.963591 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 01:47:03.963659 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 01:47:03.972220 16417 raft_consensus.cc:1261] T 0f43293f9a4f463d84ef1641d53fc3c4 P 9d953e27874844fb848a8239b3671f80 [term 1 FOLLOWER]: Refusing update from remote peer e1d7065448cd4ec0935b49302556cda5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:47:03.972224 16021 raft_consensus.cc:1261] T 0f43293f9a4f463d84ef1641d53fc3c4 P 0ac36ea9a0f54d369864a4b4789a1570 [term 1 FOLLOWER]: Refusing update from remote peer e1d7065448cd4ec0935b49302556cda5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:47:03.973225 16518 consensus_queue.cc:1035] T 0f43293f9a4f463d84ef1641d53fc3c4 P e1d7065448cd4ec0935b49302556cda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac36ea9a0f54d369864a4b4789a1570" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 42841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:47:03.973486 16525 consensus_queue.cc:1035] T 0f43293f9a4f463d84ef1641d53fc3c4 P e1d7065448cd4ec0935b49302556cda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d953e27874844fb848a8239b3671f80" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 34691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:47:03.981873 16531 mvcc.cc:205] Tried to move back new op lower bound from 7316793032579325952 to 7316793032420421632. Current Snapshot: MvccSnapshot[applied={T|T < 7316793032579325952}]
I20260810 01:47:03.984601 16532 mvcc.cc:205] Tried to move back new op lower bound from 7316793032579325952 to 7316793032420421632. Current Snapshot: MvccSnapshot[applied={T|T < 7316793032579325952}]
I20260810 01:47:03.984819 16533 mvcc.cc:205] Tried to move back new op lower bound from 7316793032579325952 to 7316793032420421632. Current Snapshot: MvccSnapshot[applied={T|T < 7316793032579325952}]
I20260810 01:47:03.993314 15816 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:47:03.993554 15816 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:47:03.996559 15816 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b4ba6db7ce24334a8f8e5e4493e20cf: Sending DeleteTablet for 3 replicas of tablet 0f43293f9a4f463d84ef1641d53fc3c4
I20260810 01:47:03.997965 15990 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f43293f9a4f463d84ef1641d53fc3c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:47:03 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:51732
I20260810 01:47:03.998631 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 15902
I20260810 01:47:03.998696 16379 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f43293f9a4f463d84ef1641d53fc3c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:47:03 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:36696
I20260810 01:47:04.011612 16188 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f43293f9a4f463d84ef1641d53fc3c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:47:03 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:48096
I20260810 01:47:04.012511 16549 tablet_replica.cc:330] T 0f43293f9a4f463d84ef1641d53fc3c4 P 9d953e27874844fb848a8239b3671f80: stopping tablet replica
I20260810 01:47:04.013038 16549 raft_consensus.cc:2229] T 0f43293f9a4f463d84ef1641d53fc3c4 P 9d953e27874844fb848a8239b3671f80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:47:04.013577 16549 raft_consensus.cc:2258] T 0f43293f9a4f463d84ef1641d53fc3c4 P 9d953e27874844fb848a8239b3671f80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:47:04.015806 16549 ts_tablet_manager.cc:1905] T 0f43293f9a4f463d84ef1641d53fc3c4 P 9d953e27874844fb848a8239b3671f80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:47:04.019106 16549 ts_tablet_manager.cc:1918] T 0f43293f9a4f463d84ef1641d53fc3c4 P 9d953e27874844fb848a8239b3671f80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:47:04.019227 16549 log.cc:1196] T 0f43293f9a4f463d84ef1641d53fc3c4 P 9d953e27874844fb848a8239b3671f80: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786326392877349-9526-0/minicluster-data/ts-2/wal/wals/0f43293f9a4f463d84ef1641d53fc3c4
W20260810 01:47:04.019534 15796 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.77.129:42841 (error 108)
I20260810 01:47:04.019620 16549 ts_tablet_manager.cc:1939] T 0f43293f9a4f463d84ef1641d53fc3c4 P 9d953e27874844fb848a8239b3671f80: Deleting consensus metadata
I20260810 01:47:04.019698 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 16101
I20260810 01:47:04.019881 16552 tablet_replica.cc:330] T 0f43293f9a4f463d84ef1641d53fc3c4 P e1d7065448cd4ec0935b49302556cda5: stopping tablet replica
I20260810 01:47:04.025904 15794 catalog_manager.cc:4841] TS 9d953e27874844fb848a8239b3671f80 (127.9.77.131:34691): tablet 0f43293f9a4f463d84ef1641d53fc3c4 (table test-table [id=4c3fc8396f664b4d889e871fe1636609]) successfully deleted
I20260810 01:47:04.026696 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:04.026259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:42841 (local address 127.0.0.1:51738)
0810 01:47:04.026451 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:04.026559 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.129:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260810 01:47:04.026952 15796 catalog_manager.cc:4584] TS 0ac36ea9a0f54d369864a4b4789a1570 (127.9.77.129:42841): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f43293f9a4f463d84ef1641d53fc3c4: Network error: Client connection negotiation failed: client connection to 127.9.77.129:42841: connect: Connection refused (error 111)
I20260810 01:47:04.033012 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 16289
I20260810 01:47:04.044693 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:04.033222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:37683 (local address 127.0.0.1:48106)
0810 01:47:04.044402 (+ 11180us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:04.044520 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.130:37683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10998}
I20260810 01:47:04.051466 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 15772
2026-08-10T01:47:04Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3026 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T01:47:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:47:04Z Disabled control of system clock
I20260810 01:47:04.093232 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:42791
--webserver_interface=127.9.77.190
--webserver_port=0
--builtin_ntp_servers=127.9.77.148:39569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:42791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 01:47:04.236658 16561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:04.237051 16561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:04.237097 16561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:04.241868 16561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:47:04.241952 16561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:04.241971 16561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:04.241991 16561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:47:04.242008 16561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:47:04.247357 16561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:39569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:42791
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:42791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:04.248776 16561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:04.250566 16561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 01:47:04.261080 16561 server_base.cc:1034] running on GCE node
W20260810 01:47:04.261030 16567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:04.261037 16566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:04.261037 16570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:04.262512 16561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:04.263866 16561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:04.265056 16561 hybrid_clock.cc:648] HybridClock initialized: now 1786326424265017 us; error 50 us; skew 500 ppm
I20260810 01:47:04.267488 16561 webserver.cc:460] Webserver started at http://127.9.77.190:46555/ using document root <none> and password file <none>
I20260810 01:47:04.268308 16561 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:04.268383 16561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:04.268651 16561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:04.271238 16561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "1736832f162e49fd9a0e56f80317eaa6"
format_stamp: "Formatted at 2026-08-10 01:47:04 on dist-test-slave-kvfs"
server_key: "6106ea12cabf2b091bc1b2e70e390963"
server_key_iv: "23e2a472d5349f1a54728a7b07d28411"
server_key_version: "encryptionkey@0"
I20260810 01:47:04.271967 16561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "1736832f162e49fd9a0e56f80317eaa6"
format_stamp: "Formatted at 2026-08-10 01:47:04 on dist-test-slave-kvfs"
server_key: "6106ea12cabf2b091bc1b2e70e390963"
server_key_iv: "23e2a472d5349f1a54728a7b07d28411"
server_key_version: "encryptionkey@0"
I20260810 01:47:04.276516 16561 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 01:47:04.279388 16583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:04.280875 16561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:47:04.281039 16561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "1736832f162e49fd9a0e56f80317eaa6"
format_stamp: "Formatted at 2026-08-10 01:47:04 on dist-test-slave-kvfs"
server_key: "6106ea12cabf2b091bc1b2e70e390963"
server_key_iv: "23e2a472d5349f1a54728a7b07d28411"
server_key_version: "encryptionkey@0"
I20260810 01:47:04.281160 16561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:04.286486 16561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:04.287209 16561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:04.287427 16561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:04.295888 16561 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:42791
I20260810 01:47:04.295991 16653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:42791 every 8 connection(s)
I20260810 01:47:04.297175 16561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:47:04.301332 16654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:04.301445 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 16561
I20260810 01:47:04.301644 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:47:04.302040 9526 external_mini_cluster.cc:1428] Setting key 4b2cc038e095012331eb98cd24132349
I20260810 01:47:04.310164 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:04.304070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53250 (local address 127.9.77.190:42791)
0810 01:47:04.305018 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:04.305042 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:04.305115 (+ 73us) server_negotiation.cc:407] Connection header received
0810 01:47:04.306082 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:04.306112 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:04.306705 (+ 593us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:04.307186 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:04.308222 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:04.308229 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:04.308247 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:47:04.308261 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:04.308343 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:47:04.308372 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:04.308555 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:04.308643 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 01:47:04.308971 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":168,"threads_started":1}
I20260810 01:47:04.310921 16654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1736832f162e49fd9a0e56f80317eaa6: Bootstrap starting.
I20260810 01:47:04.313961 16654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1736832f162e49fd9a0e56f80317eaa6: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:04.314905 16654 log.cc:824] T 00000000000000000000000000000000 P 1736832f162e49fd9a0e56f80317eaa6: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:04.317737 16654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1736832f162e49fd9a0e56f80317eaa6: No bootstrap required, opened a new log
I20260810 01:47:04.322149 16654 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1736832f162e49fd9a0e56f80317eaa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1736832f162e49fd9a0e56f80317eaa6" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 42791 } }
I20260810 01:47:04.322649 16654 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1736832f162e49fd9a0e56f80317eaa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:04.322804 16654 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1736832f162e49fd9a0e56f80317eaa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1736832f162e49fd9a0e56f80317eaa6, State: Initialized, Role: FOLLOWER
I20260810 01:47:04.323436 16654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1736832f162e49fd9a0e56f80317eaa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1736832f162e49fd9a0e56f80317eaa6" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 42791 } }
I20260810 01:47:04.323647 16654 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1736832f162e49fd9a0e56f80317eaa6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:47:04.323736 16654 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1736832f162e49fd9a0e56f80317eaa6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:47:04.323860 16654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1736832f162e49fd9a0e56f80317eaa6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:04.325143 16654 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1736832f162e49fd9a0e56f80317eaa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1736832f162e49fd9a0e56f80317eaa6" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 42791 } }
I20260810 01:47:04.325552 16654 leader_election.cc:302] T 00000000000000000000000000000000 P 1736832f162e49fd9a0e56f80317eaa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1736832f162e49fd9a0e56f80317eaa6; no voters:
I20260810 01:47:04.325899 16654 leader_election.cc:288] T 00000000000000000000000000000000 P 1736832f162e49fd9a0e56f80317eaa6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:47:04.326084 16661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1736832f162e49fd9a0e56f80317eaa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:47:04.326409 16661 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1736832f162e49fd9a0e56f80317eaa6 [term 1 LEADER]: Becoming Leader. State: Replica: 1736832f162e49fd9a0e56f80317eaa6, State: Running, Role: LEADER
I20260810 01:47:04.326805 16661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1736832f162e49fd9a0e56f80317eaa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1736832f162e49fd9a0e56f80317eaa6" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 42791 } }
I20260810 01:47:04.327602 16654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1736832f162e49fd9a0e56f80317eaa6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:47:04.329111 16662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1736832f162e49fd9a0e56f80317eaa6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1736832f162e49fd9a0e56f80317eaa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1736832f162e49fd9a0e56f80317eaa6" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 42791 } } }
I20260810 01:47:04.329097 16663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1736832f162e49fd9a0e56f80317eaa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1736832f162e49fd9a0e56f80317eaa6. Latest consensus state: current_term: 1 leader_uuid: "1736832f162e49fd9a0e56f80317eaa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1736832f162e49fd9a0e56f80317eaa6" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 42791 } } }
I20260810 01:47:04.329408 16662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1736832f162e49fd9a0e56f80317eaa6 [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:04.329408 16663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1736832f162e49fd9a0e56f80317eaa6 [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:04.330077 16675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:47:04.334434 16675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:47:04.341495 16675 catalog_manager.cc:1347] Generated new cluster ID: 8f2375929636424a8cab181a33c16e6c
I20260810 01:47:04.341588 16675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:47:04.379645 16675 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:47:04.380229 16675 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:47:04.384873 16675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1736832f162e49fd9a0e56f80317eaa6: Generated new TSK 0
I20260810 01:47:04.385895 16675 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:47:04.397351 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:0
--local_ip_for_outbound_sockets=127.9.77.129
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:42791
--builtin_ntp_servers=127.9.77.148:39569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 01:47:04.536096 16692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:04.536450 16692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:04.536496 16692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:04.540726 16692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:04.540805 16692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:04.540868 16692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:47:04.545827 16692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:39569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:42791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:04.547281 16692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:04.549274 16692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:04.560889 16702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:04.560905 16699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:04.560889 16698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:04.561326 16692 server_base.cc:1034] running on GCE node
I20260810 01:47:04.561844 16692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:04.562633 16692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:04.563845 16692 hybrid_clock.cc:648] HybridClock initialized: now 1786326424563831 us; error 37 us; skew 500 ppm
I20260810 01:47:04.566174 16692 webserver.cc:460] Webserver started at http://127.9.77.129:39915/ using document root <none> and password file <none>
I20260810 01:47:04.566980 16692 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:04.567082 16692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:04.567375 16692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:04.570045 16692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-0/data/instance:
uuid: "ae80e3327eef4344b18de2c3810cd088"
format_stamp: "Formatted at 2026-08-10 01:47:04 on dist-test-slave-kvfs"
server_key: "f557f68cc4e6347f4b1cd77a5d35f775"
server_key_iv: "6a2bfb954ec788885df88109685fec95"
server_key_version: "encryptionkey@0"
I20260810 01:47:04.570734 16692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance:
uuid: "ae80e3327eef4344b18de2c3810cd088"
format_stamp: "Formatted at 2026-08-10 01:47:04 on dist-test-slave-kvfs"
server_key: "f557f68cc4e6347f4b1cd77a5d35f775"
server_key_iv: "6a2bfb954ec788885df88109685fec95"
server_key_version: "encryptionkey@0"
I20260810 01:47:04.575376 16692 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260810 01:47:04.578510 16708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:04.580158 16692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:47:04.580328 16692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "ae80e3327eef4344b18de2c3810cd088"
format_stamp: "Formatted at 2026-08-10 01:47:04 on dist-test-slave-kvfs"
server_key: "f557f68cc4e6347f4b1cd77a5d35f775"
server_key_iv: "6a2bfb954ec788885df88109685fec95"
server_key_version: "encryptionkey@0"
I20260810 01:47:04.580471 16692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:04.586246 16692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:04.587152 16692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:04.587325 16692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:04.587940 16692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:04.589092 16692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:04.589143 16692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:04.589184 16692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:04.589200 16692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:04.599833 16692 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:43857
I20260810 01:47:04.599858 16868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:43857 every 8 connection(s)
I20260810 01:47:04.601060 16692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:47:04.607770 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 16692
I20260810 01:47:04.607985 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance
I20260810 01:47:04.608399 9526 external_mini_cluster.cc:1428] Setting key df7ddca6eecc1e556136fd50771fdd5f
I20260810 01:47:04.609117 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:04.603512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:42405 (local address 127.9.77.190:42791)
0810 01:47:04.603703 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:04.603707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:04.605156 (+ 1449us) server_negotiation.cc:407] Connection header received
0810 01:47:04.606782 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:04.606786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:04.606851 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:04.606975 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:04.608527 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:04.608534 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:04.608542 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:47:04.608559 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:04.608586 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:47:04.608596 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:04.608714 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:04.608880 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 01:47:04.608944 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 01:47:04.610435 16874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:04.604010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:42791 (local address 127.9.77.129:42405)
0810 01:47:04.604958 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:04.605008 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:04.606508 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:04.607198 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:04.607207 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:04.607321 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:04.608203 (+ 882us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 01:47:04.608204 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:04.608256 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 01:47:04.608346 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:04.608697 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:04.608708 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:04.608714 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 01:47:04.608939 (+ 225us) client_negotiation.cc:240] Negotiation successful
0810 01:47:04.609257 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":172,"threads_started":1}
I20260810 01:47:04.611171 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:0
--local_ip_for_outbound_sockets=127.9.77.130
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:42791
--builtin_ntp_servers=127.9.77.148:39569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 01:47:04.611961 16871 heartbeater.cc:347] Connected to a master server at 127.9.77.190:42791
I20260810 01:47:04.612252 16871 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:04.612777 16871 heartbeater.cc:511] Master 127.9.77.190:42791 requested a full tablet report, sending...
I20260810 01:47:04.614610 16601 ts_manager.cc:194] Registered new tserver with Master: ae80e3327eef4344b18de2c3810cd088 (127.9.77.129:43857)
W20260810 01:47:04.747740 16876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:04.748119 16876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:04.748165 16876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:04.752286 16876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:04.752351 16876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:04.752409 16876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:47:04.757308 16876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:39569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:42791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:04.758634 16876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:04.760408 16876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:04.771943 16887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:04.771899 16883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:04.771976 16885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:04.772151 16876 server_base.cc:1034] running on GCE node
I20260810 01:47:04.772764 16876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:04.773566 16876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:04.774791 16876 hybrid_clock.cc:648] HybridClock initialized: now 1786326424774758 us; error 44 us; skew 500 ppm
I20260810 01:47:04.777087 16876 webserver.cc:460] Webserver started at http://127.9.77.130:39203/ using document root <none> and password file <none>
I20260810 01:47:04.777815 16876 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:04.777917 16876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:04.778188 16876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:04.780750 16876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-1/data/instance:
uuid: "c919b042af974638946548f6d612074a"
format_stamp: "Formatted at 2026-08-10 01:47:04 on dist-test-slave-kvfs"
server_key: "eb3ef7bc6b6217c5587cb98466d55e99"
server_key_iv: "df8fe4935ab56a2f2620cee59cb7743c"
server_key_version: "encryptionkey@0"
I20260810 01:47:04.781441 16876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance:
uuid: "c919b042af974638946548f6d612074a"
format_stamp: "Formatted at 2026-08-10 01:47:04 on dist-test-slave-kvfs"
server_key: "eb3ef7bc6b6217c5587cb98466d55e99"
server_key_iv: "df8fe4935ab56a2f2620cee59cb7743c"
server_key_version: "encryptionkey@0"
I20260810 01:47:04.786293 16876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 01:47:04.789331 16895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:04.790964 16876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:47:04.791129 16876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "c919b042af974638946548f6d612074a"
format_stamp: "Formatted at 2026-08-10 01:47:04 on dist-test-slave-kvfs"
server_key: "eb3ef7bc6b6217c5587cb98466d55e99"
server_key_iv: "df8fe4935ab56a2f2620cee59cb7743c"
server_key_version: "encryptionkey@0"
I20260810 01:47:04.791265 16876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:04.796554 16876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:04.797484 16876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:04.797703 16876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:04.798350 16876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:04.799432 16876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:04.799484 16876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:04.799525 16876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:04.799566 16876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:04.810038 16876 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:38345
I20260810 01:47:04.810135 17064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:38345 every 8 connection(s)
I20260810 01:47:04.811249 16876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:47:04.811738 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 16876
I20260810 01:47:04.811928 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance
I20260810 01:47:04.812393 9526 external_mini_cluster.cc:1428] Setting key c114dd9641483def725693ae4cff74b3
I20260810 01:47:04.815490 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:0
--local_ip_for_outbound_sockets=127.9.77.131
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:42791
--builtin_ntp_servers=127.9.77.148:39569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 01:47:04.819660 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:04.813929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:38799 (local address 127.9.77.190:42791)
0810 01:47:04.814117 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:04.814123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:04.815748 (+ 1625us) server_negotiation.cc:407] Connection header received
0810 01:47:04.817529 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:04.817533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:04.817589 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:04.817704 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:04.819072 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:04.819076 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:04.819079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:47:04.819094 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:04.819130 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:47:04.819140 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:04.819245 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:04.819424 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 01:47:04.819485 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 01:47:04.820943 17068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:04.814586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:42791 (local address 127.9.77.130:38799)
0810 01:47:04.815536 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:04.815627 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:04.817275 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:04.817914 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:04.817922 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:04.818032 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:04.818767 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 01:47:04.818769 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:04.818822 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 01:47:04.818903 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:04.819235 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:04.819244 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:04.819252 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 01:47:04.819471 (+ 219us) client_negotiation.cc:240] Negotiation successful
0810 01:47:04.819871 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":184,"threads_started":1}
I20260810 01:47:04.822481 17066 heartbeater.cc:347] Connected to a master server at 127.9.77.190:42791
I20260810 01:47:04.822897 17066 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:04.823485 17066 heartbeater.cc:511] Master 127.9.77.190:42791 requested a full tablet report, sending...
I20260810 01:47:04.825048 16601 ts_manager.cc:194] Registered new tserver with Master: c919b042af974638946548f6d612074a (127.9.77.130:38345)
W20260810 01:47:04.955791 17069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:04.956244 17069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:04.956346 17069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:04.960620 17069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:04.960736 17069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:04.960839 17069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:47:04.965924 17069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:39569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:42791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:04.967525 17069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:04.969358 17069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:04.982012 17077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:04.982031 17079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:04.982012 17075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:04.982339 17069 server_base.cc:1034] running on GCE node
I20260810 01:47:04.982957 17069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:04.983752 17069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:04.985013 17069 hybrid_clock.cc:648] HybridClock initialized: now 1786326424984980 us; error 33 us; skew 500 ppm
I20260810 01:47:04.987689 17069 webserver.cc:460] Webserver started at http://127.9.77.131:37779/ using document root <none> and password file <none>
I20260810 01:47:04.988489 17069 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:04.988570 17069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:04.988807 17069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:04.991462 17069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-2/data/instance:
uuid: "5698c78fd5244317a5b64eaff6b06c18"
format_stamp: "Formatted at 2026-08-10 01:47:04 on dist-test-slave-kvfs"
server_key: "88a1b35fb30b7dea2b6c02235a91e7cf"
server_key_iv: "81e94829101b5994126be1d66de9eaf0"
server_key_version: "encryptionkey@0"
I20260810 01:47:04.992206 17069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance:
uuid: "5698c78fd5244317a5b64eaff6b06c18"
format_stamp: "Formatted at 2026-08-10 01:47:04 on dist-test-slave-kvfs"
server_key: "88a1b35fb30b7dea2b6c02235a91e7cf"
server_key_iv: "81e94829101b5994126be1d66de9eaf0"
server_key_version: "encryptionkey@0"
I20260810 01:47:04.997188 17069 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 01:47:05.000123 17085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:05.001677 17069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 01:47:05.001847 17069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "5698c78fd5244317a5b64eaff6b06c18"
format_stamp: "Formatted at 2026-08-10 01:47:04 on dist-test-slave-kvfs"
server_key: "88a1b35fb30b7dea2b6c02235a91e7cf"
server_key_iv: "81e94829101b5994126be1d66de9eaf0"
server_key_version: "encryptionkey@0"
I20260810 01:47:05.001986 17069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:05.007527 17069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:05.008374 17069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:05.008587 17069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:05.009339 17069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:05.010489 17069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:05.010577 17069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:05.010700 17069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:05.010740 17069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:05.021198 17069 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:38233
I20260810 01:47:05.021224 17250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:38233 every 8 connection(s)
I20260810 01:47:05.022346 17069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:47:05.026129 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 17069
I20260810 01:47:05.026270 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance
I20260810 01:47:05.026693 9526 external_mini_cluster.cc:1428] Setting key a28b9975992157c00146280970bbcde5
I20260810 01:47:05.030966 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:05.024748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:41261 (local address 127.9.77.190:42791)
0810 01:47:05.024961 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:05.024967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:05.026428 (+ 1461us) server_negotiation.cc:407] Connection header received
0810 01:47:05.028661 (+ 2233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:05.028665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:05.028747 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:05.028853 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:05.030411 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:05.030415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:05.030418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:47:05.030431 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:05.030464 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:47:05.030480 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:05.030581 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:05.030749 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 01:47:05.030805 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 01:47:05.032004 17257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:05.025249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:42791 (local address 127.9.77.131:41261)
0810 01:47:05.026208 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:05.026294 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:05.028328 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:05.029074 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:05.029081 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:05.029186 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:05.029995 (+ 809us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 01:47:05.029999 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:05.030088 (+ 89us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 01:47:05.030210 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:05.030568 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:05.030577 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:05.030585 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 01:47:05.030798 (+ 213us) client_negotiation.cc:240] Negotiation successful
0810 01:47:05.031083 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":146,"threads_started":1}
I20260810 01:47:05.033298 17251 heartbeater.cc:347] Connected to a master server at 127.9.77.190:42791
I20260810 01:47:05.033582 17251 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:05.034085 17251 heartbeater.cc:511] Master 127.9.77.190:42791 requested a full tablet report, sending...
I20260810 01:47:05.035287 16601 ts_manager.cc:194] Registered new tserver with Master: 5698c78fd5244317a5b64eaff6b06c18 (127.9.77.131:38233)
I20260810 01:47:05.041821 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 01:47:05.046307 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:05.044272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53254 (local address 127.9.77.190:42791)
0810 01:47:05.044429 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:05.044433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:05.044760 (+ 327us) server_negotiation.cc:407] Connection header received
0810 01:47:05.044811 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:05.044813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:05.044862 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:05.044954 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:05.045883 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:05.045885 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:05.045887 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:47:05.045900 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:05.045919 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:47:05.045927 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:05.046012 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:05.046135 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 01:47:05.046177 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 01:47:05.049805 16601 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 01:47:05.053612 16601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:47:05.068351 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:05.064783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:38345 (local address 127.0.0.1:33944)
0810 01:47:05.066025 (+ 1242us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:05.066038 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:05.066112 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:05.066805 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:05.066811 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:05.066824 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:05.067518 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 01:47:05.067521 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:05.067534 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 01:47:05.067572 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:05.067899 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:05.067911 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:05.067917 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 01:47:05.068079 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 01:47:05.068165 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1165,"thread_start_us":108,"threads_started":1}
I20260810 01:47:05.068867 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:05.065042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39320 (local address 127.9.77.129:43857)
0810 01:47:05.066132 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:05.066143 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:05.066164 (+ 21us) server_negotiation.cc:407] Connection header received
0810 01:47:05.066218 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:05.066223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:05.066374 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:05.066544 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:05.068097 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:05.068103 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:05.068112 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:47:05.068126 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:05.068166 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:47:05.068205 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:05.068542 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:05.068591 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 01:47:05.068681 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":987,"thread_start_us":207,"threads_started":1}
I20260810 01:47:05.069069 17272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:05.065666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33944 (local address 127.9.77.130:38345)
0810 01:47:05.066355 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:05.066364 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:05.066385 (+ 21us) server_negotiation.cc:407] Connection header received
0810 01:47:05.066434 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:05.066437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:05.066642 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:05.067022 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:05.067718 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:05.067724 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:05.067739 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:47:05.067754 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:05.067789 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:47:05.067808 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:05.067934 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:05.068756 (+ 822us) server_negotiation.cc:299] Negotiation successful
0810 01:47:05.068850 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":111,"threads_started":1}
I20260810 01:47:05.069109 17271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:05.065508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56940 (local address 127.9.77.131:38233)
0810 01:47:05.066013 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:05.066027 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:05.066057 (+ 30us) server_negotiation.cc:407] Connection header received
0810 01:47:05.066127 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:05.066133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:05.066376 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:05.066544 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:05.068058 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:05.068064 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:05.068075 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:47:05.068088 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:05.068128 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:47:05.068158 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:05.068274 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:05.068835 (+ 561us) server_negotiation.cc:299] Negotiation successful
0810 01:47:05.068912 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":184,"threads_started":1}
I20260810 01:47:05.068603 17266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:05.064534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:38233 (local address 127.0.0.1:56940)
0810 01:47:05.065520 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:05.065562 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:05.065692 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:05.067003 (+ 1311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:05.067006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:05.067020 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:05.067740 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 01:47:05.067742 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:05.067752 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 01:47:05.067775 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:05.068320 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:05.068324 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:05.068331 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 01:47:05.068410 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 01:47:05.068451 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"spinlock_wait_cycles":5248,"thread_start_us":131,"threads_started":1}
I20260810 01:47:05.068750 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:05.064484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:43857 (local address 127.0.0.1:39320)
0810 01:47:05.065503 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:05.065562 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:05.065692 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:05.066565 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:05.066572 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:05.066657 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:05.067463 (+ 806us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 01:47:05.067465 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:05.067491 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 01:47:05.067570 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:05.068360 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:05.068366 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:05.068373 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 01:47:05.068427 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 01:47:05.068451 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"spinlock_wait_cycles":7296,"thread_start_us":131,"threads_started":1}
I20260810 01:47:05.072860 16777 tablet_service.cc:1467] Processing CreateTablet for tablet d8595af39bfb455b9d55a0b1b2ec3481 (DEFAULT_TABLE table=test-table [id=2b84c122915e41aebed0c24d236f7a37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:47:05.072860 16973 tablet_service.cc:1467] Processing CreateTablet for tablet d8595af39bfb455b9d55a0b1b2ec3481 (DEFAULT_TABLE table=test-table [id=2b84c122915e41aebed0c24d236f7a37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:47:05.074292 16973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8595af39bfb455b9d55a0b1b2ec3481. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:05.074292 16777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8595af39bfb455b9d55a0b1b2ec3481. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:05.074129 17167 tablet_service.cc:1467] Processing CreateTablet for tablet d8595af39bfb455b9d55a0b1b2ec3481 (DEFAULT_TABLE table=test-table [id=2b84c122915e41aebed0c24d236f7a37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:47:05.075228 17167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8595af39bfb455b9d55a0b1b2ec3481. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:05.082597 17273 tablet_bootstrap.cc:492] T d8595af39bfb455b9d55a0b1b2ec3481 P ae80e3327eef4344b18de2c3810cd088: Bootstrap starting.
I20260810 01:47:05.084107 17275 tablet_bootstrap.cc:492] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18: Bootstrap starting.
I20260810 01:47:05.084107 17276 tablet_bootstrap.cc:492] T d8595af39bfb455b9d55a0b1b2ec3481 P c919b042af974638946548f6d612074a: Bootstrap starting.
I20260810 01:47:05.085649 17273 tablet_bootstrap.cc:654] T d8595af39bfb455b9d55a0b1b2ec3481 P ae80e3327eef4344b18de2c3810cd088: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:05.086629 17275 tablet_bootstrap.cc:654] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:05.086629 17276 tablet_bootstrap.cc:654] T d8595af39bfb455b9d55a0b1b2ec3481 P c919b042af974638946548f6d612074a: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:05.086730 17273 log.cc:824] T d8595af39bfb455b9d55a0b1b2ec3481 P ae80e3327eef4344b18de2c3810cd088: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:05.087586 17275 log.cc:824] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:05.087586 17276 log.cc:824] T d8595af39bfb455b9d55a0b1b2ec3481 P c919b042af974638946548f6d612074a: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:05.089356 17273 tablet_bootstrap.cc:492] T d8595af39bfb455b9d55a0b1b2ec3481 P ae80e3327eef4344b18de2c3810cd088: No bootstrap required, opened a new log
I20260810 01:47:05.089607 17273 ts_tablet_manager.cc:1397] T d8595af39bfb455b9d55a0b1b2ec3481 P ae80e3327eef4344b18de2c3810cd088: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 01:47:05.089891 17275 tablet_bootstrap.cc:492] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18: No bootstrap required, opened a new log
I20260810 01:47:05.090113 17275 ts_tablet_manager.cc:1397] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 01:47:05.090167 17276 tablet_bootstrap.cc:492] T d8595af39bfb455b9d55a0b1b2ec3481 P c919b042af974638946548f6d612074a: No bootstrap required, opened a new log
I20260810 01:47:05.090368 17276 ts_tablet_manager.cc:1397] T d8595af39bfb455b9d55a0b1b2ec3481 P c919b042af974638946548f6d612074a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 01:47:05.093804 17273 raft_consensus.cc:348] T d8595af39bfb455b9d55a0b1b2ec3481 P ae80e3327eef4344b18de2c3810cd088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae80e3327eef4344b18de2c3810cd088" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 43857 } } peers { permanent_uuid: "5698c78fd5244317a5b64eaff6b06c18" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 38233 } } peers { permanent_uuid: "c919b042af974638946548f6d612074a" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38345 } }
I20260810 01:47:05.094151 17273 raft_consensus.cc:374] T d8595af39bfb455b9d55a0b1b2ec3481 P ae80e3327eef4344b18de2c3810cd088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:05.094210 17273 raft_consensus.cc:729] T d8595af39bfb455b9d55a0b1b2ec3481 P ae80e3327eef4344b18de2c3810cd088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae80e3327eef4344b18de2c3810cd088, State: Initialized, Role: FOLLOWER
I20260810 01:47:05.094801 17273 consensus_queue.cc:260] T d8595af39bfb455b9d55a0b1b2ec3481 P ae80e3327eef4344b18de2c3810cd088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae80e3327eef4344b18de2c3810cd088" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 43857 } } peers { permanent_uuid: "5698c78fd5244317a5b64eaff6b06c18" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 38233 } } peers { permanent_uuid: "c919b042af974638946548f6d612074a" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38345 } }
I20260810 01:47:05.095141 17275 raft_consensus.cc:348] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae80e3327eef4344b18de2c3810cd088" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 43857 } } peers { permanent_uuid: "5698c78fd5244317a5b64eaff6b06c18" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 38233 } } peers { permanent_uuid: "c919b042af974638946548f6d612074a" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38345 } }
I20260810 01:47:05.095158 17276 raft_consensus.cc:348] T d8595af39bfb455b9d55a0b1b2ec3481 P c919b042af974638946548f6d612074a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae80e3327eef4344b18de2c3810cd088" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 43857 } } peers { permanent_uuid: "5698c78fd5244317a5b64eaff6b06c18" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 38233 } } peers { permanent_uuid: "c919b042af974638946548f6d612074a" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38345 } }
I20260810 01:47:05.095436 17275 raft_consensus.cc:374] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:05.095471 17276 raft_consensus.cc:374] T d8595af39bfb455b9d55a0b1b2ec3481 P c919b042af974638946548f6d612074a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:05.095504 17275 raft_consensus.cc:729] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5698c78fd5244317a5b64eaff6b06c18, State: Initialized, Role: FOLLOWER
I20260810 01:47:05.095539 17276 raft_consensus.cc:729] T d8595af39bfb455b9d55a0b1b2ec3481 P c919b042af974638946548f6d612074a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c919b042af974638946548f6d612074a, State: Initialized, Role: FOLLOWER
I20260810 01:47:05.095849 16871 heartbeater.cc:503] Master 127.9.77.190:42791 was elected leader, sending a full tablet report...
I20260810 01:47:05.096077 17275 consensus_queue.cc:260] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae80e3327eef4344b18de2c3810cd088" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 43857 } } peers { permanent_uuid: "5698c78fd5244317a5b64eaff6b06c18" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 38233 } } peers { permanent_uuid: "c919b042af974638946548f6d612074a" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38345 } }
I20260810 01:47:05.096122 17276 consensus_queue.cc:260] T d8595af39bfb455b9d55a0b1b2ec3481 P c919b042af974638946548f6d612074a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae80e3327eef4344b18de2c3810cd088" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 43857 } } peers { permanent_uuid: "5698c78fd5244317a5b64eaff6b06c18" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 38233 } } peers { permanent_uuid: "c919b042af974638946548f6d612074a" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38345 } }
I20260810 01:47:05.096189 17273 ts_tablet_manager.cc:1428] T d8595af39bfb455b9d55a0b1b2ec3481 P ae80e3327eef4344b18de2c3810cd088: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 01:47:05.097116 17275 ts_tablet_manager.cc:1428] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 01:47:05.097116 17276 ts_tablet_manager.cc:1428] T d8595af39bfb455b9d55a0b1b2ec3481 P c919b042af974638946548f6d612074a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 01:47:05.097497 17066 heartbeater.cc:503] Master 127.9.77.190:42791 was elected leader, sending a full tablet report...
I20260810 01:47:05.097548 17251 heartbeater.cc:503] Master 127.9.77.190:42791 was elected leader, sending a full tablet report...
W20260810 01:47:05.102572 16872 tablet.cc:2369] T d8595af39bfb455b9d55a0b1b2ec3481 P ae80e3327eef4344b18de2c3810cd088: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 01:47:05.273633 17253 tablet.cc:2369] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 01:47:05.312968 17067 tablet.cc:2369] T d8595af39bfb455b9d55a0b1b2ec3481 P c919b042af974638946548f6d612074a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 01:47:05.433068 17286 raft_consensus.cc:482] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:47:05.433357 17286 raft_consensus.cc:504] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae80e3327eef4344b18de2c3810cd088" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 43857 } } peers { permanent_uuid: "5698c78fd5244317a5b64eaff6b06c18" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 38233 } } peers { permanent_uuid: "c919b042af974638946548f6d612074a" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38345 } }
I20260810 01:47:05.434891 17286 leader_election.cc:288] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae80e3327eef4344b18de2c3810cd088 (127.9.77.129:43857), c919b042af974638946548f6d612074a (127.9.77.130:38345)
I20260810 01:47:05.437252 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:05.435306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:41071 (local address 127.9.77.129:43857)
0810 01:47:05.435469 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:05.435474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:05.435490 (+ 16us) server_negotiation.cc:407] Connection header received
0810 01:47:05.435657 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:05.435660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:05.435708 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:05.435829 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:05.436750 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:05.436756 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:05.436760 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:47:05.436775 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:05.436805 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:47:05.436816 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:05.436924 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:05.437036 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 01:47:05.437089 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 01:47:05.437252 17272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:05.435385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:34479 (local address 127.9.77.130:38345)
0810 01:47:05.435544 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:05.435549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:05.435791 (+ 242us) server_negotiation.cc:407] Connection header received
0810 01:47:05.435857 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:05.435859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:05.435907 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:05.435985 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:05.436825 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:05.436829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:05.436834 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:47:05.436851 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:05.436878 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:47:05.436891 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:05.437008 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:05.437077 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 01:47:05.437118 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 01:47:05.437258 17257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:05.435153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:43857 (local address 127.9.77.131:41071)
0810 01:47:05.435385 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:05.435404 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:05.435504 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:05.435876 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:05.435878 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:05.435892 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:05.436522 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 01:47:05.436526 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:05.436545 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 01:47:05.436587 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:05.436910 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:05.436913 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:05.436915 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 01:47:05.437036 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 01:47:05.437082 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260810 01:47:05.437271 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:05.435248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:38345 (local address 127.9.77.131:34479)
0810 01:47:05.435670 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:05.435680 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:05.435759 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:05.436051 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:05.436054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:05.436068 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:05.436614 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 01:47:05.436618 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:05.436629 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 01:47:05.436657 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:05.436978 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:05.436983 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:05.436988 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 01:47:05.437059 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 01:47:05.437082 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":202,"threads_started":1}
I20260810 01:47:05.438068 17000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8595af39bfb455b9d55a0b1b2ec3481" candidate_uuid: "5698c78fd5244317a5b64eaff6b06c18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c919b042af974638946548f6d612074a" is_pre_election: true
I20260810 01:47:05.438062 16809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8595af39bfb455b9d55a0b1b2ec3481" candidate_uuid: "5698c78fd5244317a5b64eaff6b06c18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae80e3327eef4344b18de2c3810cd088" is_pre_election: true
I20260810 01:47:05.438448 16809 raft_consensus.cc:2454] T d8595af39bfb455b9d55a0b1b2ec3481 P ae80e3327eef4344b18de2c3810cd088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5698c78fd5244317a5b64eaff6b06c18 in term 0.
I20260810 01:47:05.438448 17000 raft_consensus.cc:2454] T d8595af39bfb455b9d55a0b1b2ec3481 P c919b042af974638946548f6d612074a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5698c78fd5244317a5b64eaff6b06c18 in term 0.
I20260810 01:47:05.439059 17086 leader_election.cc:302] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5698c78fd5244317a5b64eaff6b06c18, ae80e3327eef4344b18de2c3810cd088, c919b042af974638946548f6d612074a; no voters:
I20260810 01:47:05.439339 17286 raft_consensus.cc:2781] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:47:05.439441 17286 raft_consensus.cc:482] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:47:05.439504 17286 raft_consensus.cc:3037] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:05.440832 17286 raft_consensus.cc:504] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae80e3327eef4344b18de2c3810cd088" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 43857 } } peers { permanent_uuid: "5698c78fd5244317a5b64eaff6b06c18" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 38233 } } peers { permanent_uuid: "c919b042af974638946548f6d612074a" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38345 } }
I20260810 01:47:05.441223 17286 leader_election.cc:288] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18 [CANDIDATE]: Term 1 election: Requested vote from peers ae80e3327eef4344b18de2c3810cd088 (127.9.77.129:43857), c919b042af974638946548f6d612074a (127.9.77.130:38345)
I20260810 01:47:05.441664 17000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8595af39bfb455b9d55a0b1b2ec3481" candidate_uuid: "5698c78fd5244317a5b64eaff6b06c18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c919b042af974638946548f6d612074a"
I20260810 01:47:05.441663 16809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8595af39bfb455b9d55a0b1b2ec3481" candidate_uuid: "5698c78fd5244317a5b64eaff6b06c18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae80e3327eef4344b18de2c3810cd088"
I20260810 01:47:05.441825 16809 raft_consensus.cc:3037] T d8595af39bfb455b9d55a0b1b2ec3481 P ae80e3327eef4344b18de2c3810cd088 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:05.441828 17000 raft_consensus.cc:3037] T d8595af39bfb455b9d55a0b1b2ec3481 P c919b042af974638946548f6d612074a [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:05.443208 16809 raft_consensus.cc:2454] T d8595af39bfb455b9d55a0b1b2ec3481 P ae80e3327eef4344b18de2c3810cd088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5698c78fd5244317a5b64eaff6b06c18 in term 1.
I20260810 01:47:05.443217 17000 raft_consensus.cc:2454] T d8595af39bfb455b9d55a0b1b2ec3481 P c919b042af974638946548f6d612074a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5698c78fd5244317a5b64eaff6b06c18 in term 1.
I20260810 01:47:05.443569 17086 leader_election.cc:302] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5698c78fd5244317a5b64eaff6b06c18, c919b042af974638946548f6d612074a; no voters:
I20260810 01:47:05.443837 17286 raft_consensus.cc:2781] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:47:05.444190 17286 raft_consensus.cc:686] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18 [term 1 LEADER]: Becoming Leader. State: Replica: 5698c78fd5244317a5b64eaff6b06c18, State: Running, Role: LEADER
I20260810 01:47:05.444608 17286 consensus_queue.cc:237] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae80e3327eef4344b18de2c3810cd088" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 43857 } } peers { permanent_uuid: "5698c78fd5244317a5b64eaff6b06c18" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 38233 } } peers { permanent_uuid: "c919b042af974638946548f6d612074a" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38345 } }
I20260810 01:47:05.448652 16600 catalog_manager.cc:5495] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5698c78fd5244317a5b64eaff6b06c18 (127.9.77.131). New cstate: current_term: 1 leader_uuid: "5698c78fd5244317a5b64eaff6b06c18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae80e3327eef4344b18de2c3810cd088" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5698c78fd5244317a5b64eaff6b06c18" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 38233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c919b042af974638946548f6d612074a" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38345 } health_report { overall_health: UNKNOWN } } }
I20260810 01:47:05.526786 17271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:05.525021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56956 (local address 127.9.77.131:38233)
0810 01:47:05.525189 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:05.525194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:05.525212 (+ 18us) server_negotiation.cc:407] Connection header received
0810 01:47:05.525319 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:05.525322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:05.525416 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:05.525508 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:05.526355 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:05.526358 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:05.526360 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:47:05.526375 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:05.526401 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:47:05.526410 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:05.526496 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:05.526576 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 01:47:05.526616 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 01:47:05.535171 16809 raft_consensus.cc:1261] T d8595af39bfb455b9d55a0b1b2ec3481 P ae80e3327eef4344b18de2c3810cd088 [term 1 FOLLOWER]: Refusing update from remote peer 5698c78fd5244317a5b64eaff6b06c18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:47:05.535171 17000 raft_consensus.cc:1261] T d8595af39bfb455b9d55a0b1b2ec3481 P c919b042af974638946548f6d612074a [term 1 FOLLOWER]: Refusing update from remote peer 5698c78fd5244317a5b64eaff6b06c18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:47:05.536227 17293 consensus_queue.cc:1035] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c919b042af974638946548f6d612074a" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:47:05.536485 17286 consensus_queue.cc:1035] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae80e3327eef4344b18de2c3810cd088" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:47:05.544677 17297 mvcc.cc:205] Tried to move back new op lower bound from 7316793038981029888 to 7316793038624374784. Current Snapshot: MvccSnapshot[applied={T|T < 7316793038981029888}]
I20260810 01:47:05.546011 17302 mvcc.cc:205] Tried to move back new op lower bound from 7316793038981029888 to 7316793038624374784. Current Snapshot: MvccSnapshot[applied={T|T < 7316793038981029888}]
I20260810 01:47:05.557682 16601 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:47:05.557964 16601 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:47:05.561354 16601 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1736832f162e49fd9a0e56f80317eaa6: Sending DeleteTablet for 3 replicas of tablet d8595af39bfb455b9d55a0b1b2ec3481
I20260810 01:47:05.562510 17167 tablet_service.cc:1514] Processing DeleteTablet for tablet d8595af39bfb455b9d55a0b1b2ec3481 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:47:05 UTC) from {username='slave'} at 127.0.0.1:56940
I20260810 01:47:05.562654 16777 tablet_service.cc:1514] Processing DeleteTablet for tablet d8595af39bfb455b9d55a0b1b2ec3481 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:47:05 UTC) from {username='slave'} at 127.0.0.1:39320
I20260810 01:47:05.562734 16973 tablet_service.cc:1514] Processing DeleteTablet for tablet d8595af39bfb455b9d55a0b1b2ec3481 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:47:05 UTC) from {username='slave'} at 127.0.0.1:33944
I20260810 01:47:05.563373 17312 tablet_replica.cc:330] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18: stopping tablet replica
I20260810 01:47:05.563373 17313 tablet_replica.cc:330] T d8595af39bfb455b9d55a0b1b2ec3481 P ae80e3327eef4344b18de2c3810cd088: stopping tablet replica
I20260810 01:47:05.563817 17313 raft_consensus.cc:2229] T d8595af39bfb455b9d55a0b1b2ec3481 P ae80e3327eef4344b18de2c3810cd088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:47:05.563942 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 16692
I20260810 01:47:05.571290 17312 raft_consensus.cc:2229] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18 [term 1 LEADER]: Raft consensus shutting down.
I20260810 01:47:05.571933 17312 raft_consensus.cc:2258] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:47:05.572479 17314 tablet_replica.cc:330] T d8595af39bfb455b9d55a0b1b2ec3481 P c919b042af974638946548f6d612074a: stopping tablet replica
I20260810 01:47:05.572947 17314 raft_consensus.cc:2229] T d8595af39bfb455b9d55a0b1b2ec3481 P c919b042af974638946548f6d612074a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:47:05.573341 17314 raft_consensus.cc:2258] T d8595af39bfb455b9d55a0b1b2ec3481 P c919b042af974638946548f6d612074a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:47:05.575459 17314 ts_tablet_manager.cc:1905] T d8595af39bfb455b9d55a0b1b2ec3481 P c919b042af974638946548f6d612074a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:47:05.575459 17312 ts_tablet_manager.cc:1905] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:47:05.578934 17314 ts_tablet_manager.cc:1918] T d8595af39bfb455b9d55a0b1b2ec3481 P c919b042af974638946548f6d612074a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:47:05.579066 17314 log.cc:1196] T d8595af39bfb455b9d55a0b1b2ec3481 P c919b042af974638946548f6d612074a: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-1/wal/wals/d8595af39bfb455b9d55a0b1b2ec3481
I20260810 01:47:05.579427 17314 ts_tablet_manager.cc:1939] T d8595af39bfb455b9d55a0b1b2ec3481 P c919b042af974638946548f6d612074a: Deleting consensus metadata
I20260810 01:47:05.580062 17312 ts_tablet_manager.cc:1918] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:47:05.580273 17312 log.cc:1196] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786326392877349-9526-0/minicluster-data/ts-2/wal/wals/d8595af39bfb455b9d55a0b1b2ec3481
I20260810 01:47:05.580704 17312 ts_tablet_manager.cc:1939] T d8595af39bfb455b9d55a0b1b2ec3481 P 5698c78fd5244317a5b64eaff6b06c18: Deleting consensus metadata
I20260810 01:47:05.581283 16584 catalog_manager.cc:4841] TS c919b042af974638946548f6d612074a (127.9.77.130:38345): tablet d8595af39bfb455b9d55a0b1b2ec3481 (table test-table [id=2b84c122915e41aebed0c24d236f7a37]) successfully deleted
I20260810 01:47:05.582204 16585 catalog_manager.cc:4841] TS 5698c78fd5244317a5b64eaff6b06c18 (127.9.77.131:38233): tablet d8595af39bfb455b9d55a0b1b2ec3481 (table test-table [id=2b84c122915e41aebed0c24d236f7a37]) successfully deleted
W20260810 01:47:05.584869 16587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.77.129:43857 (error 108)
I20260810 01:47:05.585965 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 16876
I20260810 01:47:05.586390 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:05.585851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:43857 (local address 127.0.0.1:39336)
0810 01:47:05.586137 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:05.586238 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.129:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":72,"threads_started":1}
W20260810 01:47:05.587038 16587 catalog_manager.cc:4584] TS ae80e3327eef4344b18de2c3810cd088 (127.9.77.129:43857): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d8595af39bfb455b9d55a0b1b2ec3481: Network error: Client connection negotiation failed: client connection to 127.9.77.129:43857: connect: Connection refused (error 111)
I20260810 01:47:05.605058 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 17069
I20260810 01:47:05.624662 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 16561
2026-08-10T01:47:05Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1577 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 01:47:05.659250 9526 mini_oidc.cc:152] Starting JWKS server
I20260810 01:47:05.661540 9526 webserver.cc:460] Webserver started at https://127.0.0.1:38703/ using document root <none> and password file <none>
I20260810 01:47:05.661698 9526 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 01:47:05.661955 9526 webserver.cc:460] Webserver started at http://127.0.0.1:37911/ using document root <none> and password file <none>
2026-08-10T01:47:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:47:05Z Disabled control of system clock
I20260810 01:47:05.674566 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:38917
--webserver_interface=127.9.77.190
--webserver_port=0
--builtin_ntp_servers=127.9.77.148:33837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:38917
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38703/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786326392877349-9526-0/testchainca.cert with env {}
W20260810 01:47:05.809545 17332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:05.809942 17332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:05.809989 17332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:05.814323 17332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:47:05.814393 17332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:05.814409 17332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:05.814427 17332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:47:05.814442 17332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 01:47:05.814459 17332 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 01:47:05.814472 17332 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38703/jwks.json
I20260810 01:47:05.820550 17332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:33837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:38917
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:38917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38703/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786326392877349-9526-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:05.822010 17332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:05.823267 17332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:05.833485 17342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:05.833489 17341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:05.833595 17332 server_base.cc:1034] running on GCE node
W20260810 01:47:05.833475 17349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:05.834602 17332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:05.836089 17332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:05.837312 17332 hybrid_clock.cc:648] HybridClock initialized: now 1786326425837286 us; error 41 us; skew 500 ppm
I20260810 01:47:05.845597 17332 webserver.cc:460] Webserver started at http://127.9.77.190:41037/ using document root <none> and password file <none>
I20260810 01:47:05.846444 17332 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:05.846554 17332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:05.846861 17332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:05.849592 17332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "750f3dc336104453bcf7345e4f0e8b72"
format_stamp: "Formatted at 2026-08-10 01:47:05 on dist-test-slave-kvfs"
server_key: "09ef35e356d420babccbfe4a1101e9dc"
server_key_iv: "b09859078d047ef67f556c36faef0bfd"
server_key_version: "encryptionkey@0"
I20260810 01:47:05.850296 17332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "750f3dc336104453bcf7345e4f0e8b72"
format_stamp: "Formatted at 2026-08-10 01:47:05 on dist-test-slave-kvfs"
server_key: "09ef35e356d420babccbfe4a1101e9dc"
server_key_iv: "b09859078d047ef67f556c36faef0bfd"
server_key_version: "encryptionkey@0"
I20260810 01:47:05.854941 17332 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 01:47:05.857999 17359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:05.859781 17332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 01:47:05.859985 17332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "750f3dc336104453bcf7345e4f0e8b72"
format_stamp: "Formatted at 2026-08-10 01:47:05 on dist-test-slave-kvfs"
server_key: "09ef35e356d420babccbfe4a1101e9dc"
server_key_iv: "b09859078d047ef67f556c36faef0bfd"
server_key_version: "encryptionkey@0"
I20260810 01:47:05.860179 17332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:05.896979 17332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:05.898087 17332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:05.898370 17332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:05.907867 17332 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:38917
I20260810 01:47:05.907910 17432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:38917 every 8 connection(s)
I20260810 01:47:05.909186 17332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:47:05.913086 17434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:05.913300 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 17332
I20260810 01:47:05.913445 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:47:05.913854 9526 external_mini_cluster.cc:1428] Setting key 23c51fc97cfe0a9096e1d4603b2bc3f6
I20260810 01:47:05.922159 17434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 750f3dc336104453bcf7345e4f0e8b72: Bootstrap starting.
I20260810 01:47:05.925441 17434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 750f3dc336104453bcf7345e4f0e8b72: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:05.925608 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:05.915601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50580 (local address 127.9.77.190:38917)
0810 01:47:05.916451 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:05.916473 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:05.916521 (+ 48us) server_negotiation.cc:407] Connection header received
0810 01:47:05.917361 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:05.917383 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:05.917903 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:05.918308 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:05.918743 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:05.920087 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:05.921454 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:05.922036 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:05.922201 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:05.923306 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:05.923325 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:05.923341 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:47:05.923849 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:05.923920 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:47:05.923948 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:05.924112 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:05.924245 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 01:47:05.924623 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":193,"threads_started":1}
I20260810 01:47:05.926442 17434 log.cc:824] T 00000000000000000000000000000000 P 750f3dc336104453bcf7345e4f0e8b72: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:05.929090 17434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 750f3dc336104453bcf7345e4f0e8b72: No bootstrap required, opened a new log
I20260810 01:47:05.929224 9526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 01:47:05.933480 17434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 750f3dc336104453bcf7345e4f0e8b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "750f3dc336104453bcf7345e4f0e8b72" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 38917 } }
I20260810 01:47:05.933761 17434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 750f3dc336104453bcf7345e4f0e8b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:05.933852 17434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 750f3dc336104453bcf7345e4f0e8b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 750f3dc336104453bcf7345e4f0e8b72, State: Initialized, Role: FOLLOWER
I20260810 01:47:05.934358 17434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 750f3dc336104453bcf7345e4f0e8b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "750f3dc336104453bcf7345e4f0e8b72" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 38917 } }
I20260810 01:47:05.934518 17434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 750f3dc336104453bcf7345e4f0e8b72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:47:05.934588 17434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 750f3dc336104453bcf7345e4f0e8b72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:47:05.934718 17434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 750f3dc336104453bcf7345e4f0e8b72 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:05.935895 17434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 750f3dc336104453bcf7345e4f0e8b72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "750f3dc336104453bcf7345e4f0e8b72" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 38917 } }
I20260810 01:47:05.936324 17434 leader_election.cc:302] T 00000000000000000000000000000000 P 750f3dc336104453bcf7345e4f0e8b72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 750f3dc336104453bcf7345e4f0e8b72; no voters:
I20260810 01:47:05.936743 17434 leader_election.cc:288] T 00000000000000000000000000000000 P 750f3dc336104453bcf7345e4f0e8b72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:47:05.936842 17444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 750f3dc336104453bcf7345e4f0e8b72 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:47:05.937053 17444 raft_consensus.cc:686] T 00000000000000000000000000000000 P 750f3dc336104453bcf7345e4f0e8b72 [term 1 LEADER]: Becoming Leader. State: Replica: 750f3dc336104453bcf7345e4f0e8b72, State: Running, Role: LEADER
I20260810 01:47:05.937386 17444 consensus_queue.cc:237] T 00000000000000000000000000000000 P 750f3dc336104453bcf7345e4f0e8b72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "750f3dc336104453bcf7345e4f0e8b72" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 38917 } }
I20260810 01:47:05.938340 17434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 750f3dc336104453bcf7345e4f0e8b72 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:47:05.939433 17445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 750f3dc336104453bcf7345e4f0e8b72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 750f3dc336104453bcf7345e4f0e8b72. Latest consensus state: current_term: 1 leader_uuid: "750f3dc336104453bcf7345e4f0e8b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "750f3dc336104453bcf7345e4f0e8b72" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 38917 } } }
I20260810 01:47:05.939582 17445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 750f3dc336104453bcf7345e4f0e8b72 [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:05.939914 17447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 750f3dc336104453bcf7345e4f0e8b72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "750f3dc336104453bcf7345e4f0e8b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "750f3dc336104453bcf7345e4f0e8b72" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 38917 } } }
I20260810 01:47:05.940008 17447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 750f3dc336104453bcf7345e4f0e8b72 [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:05.940330 17456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:47:05.944818 17456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:47:05.951721 17456 catalog_manager.cc:1347] Generated new cluster ID: 41e8630e66f34e8a94a7e74a8897614c
I20260810 01:47:05.951830 17456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:47:05.977737 17456 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:47:05.978976 17456 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:47:05.995478 17456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 750f3dc336104453bcf7345e4f0e8b72: Generated new TSK 0
I20260810 01:47:05.996528 17456 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 01:47:06.030674 17482 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 01:47:06.030915 17482 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 01:47:06.031562 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:06.011759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50594 (local address 127.9.77.190:38917)
0810 01:47:06.011940 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:06.011946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:06.012246 (+ 300us) server_negotiation.cc:407] Connection header received
0810 01:47:06.012512 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:06.012516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:06.012586 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:06.012712 (+ 126us) server_negotiation.cc:226] Negotiated authn=JWT
0810 01:47:06.013013 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:06.013918 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:06.014876 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:06.015194 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:06.015310 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:06.015672 (+ 362us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 01:47:06.031010 (+ 15338us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38703/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 01:47:06.031375 (+ 365us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50594: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38703/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":69}
W20260810 01:47:06.031659 17438 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50594: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38703/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 01:47:06.033224 17481 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:06.011611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:38917 (local address 127.0.0.1:50594)
0810 01:47:06.012146 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:06.012161 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:06.012315 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:06.012740 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:06.012743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:06.012752 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0810 01:47:06.012884 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:06.012890 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:06.014066 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:06.014069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:06.014756 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:06.014765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:06.015315 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:06.015317 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:06.015389 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:06.015444 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 01:47:06.031311 (+ 15867us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38703/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 01:47:06.031579 (+ 268us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.77.190:38917: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38703/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":169,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.9.77.190:38917: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38703/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 01:47:06.034329 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 17332
2026-08-10T01:47:06Z chronyd exiting
I20260810 01:47:06.084152 9526 test_util.cc:175] -----------------------------------------------
I20260810 01:47:06.084287 9526 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786326392877349-9526-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (427 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 01:47:06.085866 9526 mini_oidc.cc:152] Starting JWKS server
I20260810 01:47:06.086946 9526 webserver.cc:460] Webserver started at https://127.0.0.1:44013/ using document root <none> and password file <none>
I20260810 01:47:06.087072 9526 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 01:47:06.087383 9526 webserver.cc:460] Webserver started at http://127.0.0.1:46175/ using document root <none> and password file <none>
2026-08-10T01:47:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:47:06Z Disabled control of system clock
I20260810 01:47:06.100484 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:39609
--webserver_interface=127.9.77.190
--webserver_port=0
--builtin_ntp_servers=127.9.77.148:34937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:39609
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44013/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786326392877349-9526-0/testchainca.cert with env {}
W20260810 01:47:06.237993 17489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:06.238363 17489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:06.238406 17489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:06.242761 17489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:47:06.242843 17489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:06.242859 17489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:06.242878 17489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:47:06.242894 17489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 01:47:06.242911 17489 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 01:47:06.242925 17489 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44013/jwks.json
I20260810 01:47:06.249130 17489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:34937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:39609
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:39609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44013/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786326392877349-9526-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:06.250583 17489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:06.251874 17489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:06.262557 17499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:06.262598 17497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:06.262666 17489 server_base.cc:1034] running on GCE node
W20260810 01:47:06.262576 17496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:06.263690 17489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:06.265270 17489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:06.266536 17489 hybrid_clock.cc:648] HybridClock initialized: now 1786326426266512 us; error 48 us; skew 500 ppm
I20260810 01:47:06.269177 17489 webserver.cc:460] Webserver started at http://127.9.77.190:38889/ using document root <none> and password file <none>
I20260810 01:47:06.270196 17489 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:06.270306 17489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:06.270547 17489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:06.273249 17489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "dcd6c66f6b6d4c1ba659b702915b18a5"
format_stamp: "Formatted at 2026-08-10 01:47:06 on dist-test-slave-kvfs"
server_key: "5bb3561268fade21851265a6169255eb"
server_key_iv: "4ff3bbbf70b19ce9260b8a42192f1d12"
server_key_version: "encryptionkey@0"
I20260810 01:47:06.273905 17489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "dcd6c66f6b6d4c1ba659b702915b18a5"
format_stamp: "Formatted at 2026-08-10 01:47:06 on dist-test-slave-kvfs"
server_key: "5bb3561268fade21851265a6169255eb"
server_key_iv: "4ff3bbbf70b19ce9260b8a42192f1d12"
server_key_version: "encryptionkey@0"
I20260810 01:47:06.279363 17489 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 01:47:06.282524 17506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:06.284240 17489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:47:06.284399 17489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "dcd6c66f6b6d4c1ba659b702915b18a5"
format_stamp: "Formatted at 2026-08-10 01:47:06 on dist-test-slave-kvfs"
server_key: "5bb3561268fade21851265a6169255eb"
server_key_iv: "4ff3bbbf70b19ce9260b8a42192f1d12"
server_key_version: "encryptionkey@0"
I20260810 01:47:06.284546 17489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:06.301337 17489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:06.302313 17489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:06.302582 17489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:06.312839 17489 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:39609
I20260810 01:47:06.312872 17585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:39609 every 8 connection(s)
I20260810 01:47:06.314335 17489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:47:06.318887 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 17489
I20260810 01:47:06.318995 17589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:06.319069 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:47:06.319453 9526 external_mini_cluster.cc:1428] Setting key 71997c3842d0f40baf384f8c3cb87fc1
I20260810 01:47:06.328637 17589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcd6c66f6b6d4c1ba659b702915b18a5: Bootstrap starting.
I20260810 01:47:06.331755 17594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:06.321482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39976 (local address 127.9.77.190:39609)
0810 01:47:06.322319 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:06.322339 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:06.322385 (+ 46us) server_negotiation.cc:407] Connection header received
0810 01:47:06.323385 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:06.323409 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:06.323949 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:06.324409 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:06.324857 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:06.325951 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:06.327431 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:06.327976 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:06.328229 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:06.329515 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:06.329521 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:06.329536 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:47:06.330036 (+ 500us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:06.330106 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:47:06.330131 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:06.330215 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:06.330442 (+ 227us) server_negotiation.cc:299] Negotiation successful
0810 01:47:06.330806 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":193,"threads_started":1}
I20260810 01:47:06.332188 17589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcd6c66f6b6d4c1ba659b702915b18a5: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:06.333184 17589 log.cc:824] T 00000000000000000000000000000000 P dcd6c66f6b6d4c1ba659b702915b18a5: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:06.335282 9526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 01:47:06.335978 17589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcd6c66f6b6d4c1ba659b702915b18a5: No bootstrap required, opened a new log
I20260810 01:47:06.340754 17589 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcd6c66f6b6d4c1ba659b702915b18a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcd6c66f6b6d4c1ba659b702915b18a5" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 39609 } }
I20260810 01:47:06.341111 17589 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcd6c66f6b6d4c1ba659b702915b18a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:06.341207 17589 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcd6c66f6b6d4c1ba659b702915b18a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcd6c66f6b6d4c1ba659b702915b18a5, State: Initialized, Role: FOLLOWER
I20260810 01:47:06.341830 17589 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcd6c66f6b6d4c1ba659b702915b18a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcd6c66f6b6d4c1ba659b702915b18a5" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 39609 } }
I20260810 01:47:06.342036 17589 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcd6c66f6b6d4c1ba659b702915b18a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:47:06.342131 17589 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcd6c66f6b6d4c1ba659b702915b18a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:47:06.342281 17589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcd6c66f6b6d4c1ba659b702915b18a5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:06.343781 17589 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcd6c66f6b6d4c1ba659b702915b18a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcd6c66f6b6d4c1ba659b702915b18a5" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 39609 } }
I20260810 01:47:06.344293 17589 leader_election.cc:302] T 00000000000000000000000000000000 P dcd6c66f6b6d4c1ba659b702915b18a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcd6c66f6b6d4c1ba659b702915b18a5; no voters:
I20260810 01:47:06.344707 17589 leader_election.cc:288] T 00000000000000000000000000000000 P dcd6c66f6b6d4c1ba659b702915b18a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:47:06.344905 17599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcd6c66f6b6d4c1ba659b702915b18a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:47:06.345214 17599 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcd6c66f6b6d4c1ba659b702915b18a5 [term 1 LEADER]: Becoming Leader. State: Replica: dcd6c66f6b6d4c1ba659b702915b18a5, State: Running, Role: LEADER
I20260810 01:47:06.345580 17599 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcd6c66f6b6d4c1ba659b702915b18a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcd6c66f6b6d4c1ba659b702915b18a5" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 39609 } }
I20260810 01:47:06.346410 17589 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcd6c66f6b6d4c1ba659b702915b18a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:47:06.347764 17600 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcd6c66f6b6d4c1ba659b702915b18a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcd6c66f6b6d4c1ba659b702915b18a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcd6c66f6b6d4c1ba659b702915b18a5" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 39609 } } }
I20260810 01:47:06.347957 17600 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcd6c66f6b6d4c1ba659b702915b18a5 [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:06.348299 17603 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcd6c66f6b6d4c1ba659b702915b18a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcd6c66f6b6d4c1ba659b702915b18a5. Latest consensus state: current_term: 1 leader_uuid: "dcd6c66f6b6d4c1ba659b702915b18a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcd6c66f6b6d4c1ba659b702915b18a5" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 39609 } } }
I20260810 01:47:06.348425 17603 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcd6c66f6b6d4c1ba659b702915b18a5 [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:06.348558 17614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:47:06.352669 17614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:47:06.359840 17614 catalog_manager.cc:1347] Generated new cluster ID: c836e388308a4a4685e249bd8c66f4fc
I20260810 01:47:06.359953 17614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:47:06.377884 17614 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:47:06.379508 17614 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:47:06.387749 17614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcd6c66f6b6d4c1ba659b702915b18a5: Generated new TSK 0
I20260810 01:47:06.388923 17614 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 01:47:06.419713 17634 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 01:47:06.419889 17634 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 01:47:06.420929 17594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:06.402721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39980 (local address 127.9.77.190:39609)
0810 01:47:06.402903 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:06.402908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:06.403116 (+ 208us) server_negotiation.cc:407] Connection header received
0810 01:47:06.403329 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:06.403332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:06.403403 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:06.403554 (+ 151us) server_negotiation.cc:226] Negotiated authn=JWT
0810 01:47:06.403851 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:06.404808 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:06.405897 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:06.406266 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:06.406383 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:06.406791 (+ 408us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 01:47:06.420296 (+ 13505us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44013/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 01:47:06.420702 (+ 406us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39980: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44013/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":71}
W20260810 01:47:06.421030 17594 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39980: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44013/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 01:47:06.421167 17633 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:06.402534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:39609 (local address 127.0.0.1:39980)
0810 01:47:06.403004 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:06.403020 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:06.403156 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:06.403583 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:06.403586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:06.403595 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0810 01:47:06.403717 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:06.403723 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:06.404963 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:06.404968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:06.405751 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:06.405762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:06.406422 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:06.406425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:06.406526 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:06.406555 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 01:47:06.420636 (+ 14081us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44013/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 01:47:06.420922 (+ 286us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.77.190:39609: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44013/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":162,"threads_started":1}
I20260810 01:47:06.422339 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 17489
2026-08-10T01:47:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (390 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 01:47:06.475924 9526 mini_oidc.cc:152] Starting JWKS server
I20260810 01:47:06.477039 9526 webserver.cc:460] Webserver started at https://127.0.0.1:41565/ using document root <none> and password file <none>
I20260810 01:47:06.477180 9526 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 01:47:06.477473 9526 webserver.cc:460] Webserver started at http://127.0.0.1:38611/ using document root <none> and password file <none>
2026-08-10T01:47:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:47:06Z Disabled control of system clock
I20260810 01:47:06.490025 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:42881
--webserver_interface=127.9.77.190
--webserver_port=0
--builtin_ntp_servers=127.9.77.148:44899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:42881
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41565/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 01:47:06.624797 17648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:06.625191 17648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:06.625241 17648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:06.629554 17648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:47:06.629643 17648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:06.629660 17648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:06.629678 17648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:47:06.629694 17648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 01:47:06.629711 17648 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 01:47:06.629725 17648 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41565/jwks.json
I20260810 01:47:06.634920 17648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:44899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:42881
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:42881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41565/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:06.636544 17648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:06.638523 17648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:06.649063 17656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:06.649067 17655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:06.649067 17658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:06.649525 17648 server_base.cc:1034] running on GCE node
I20260810 01:47:06.650326 17648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:06.651804 17648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:06.653033 17648 hybrid_clock.cc:648] HybridClock initialized: now 1786326426652994 us; error 57 us; skew 500 ppm
I20260810 01:47:06.655555 17648 webserver.cc:460] Webserver started at http://127.9.77.190:32973/ using document root <none> and password file <none>
I20260810 01:47:06.656443 17648 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:06.656556 17648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:06.656858 17648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:06.659745 17648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "4ca61933fb3a4dd89a0ff3c7ea581535"
format_stamp: "Formatted at 2026-08-10 01:47:06 on dist-test-slave-kvfs"
server_key: "8bfc8c537274497366aad092125cb4b9"
server_key_iv: "48db9c9d9e6aa99883b262139d1679cf"
server_key_version: "encryptionkey@0"
I20260810 01:47:06.660568 17648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "4ca61933fb3a4dd89a0ff3c7ea581535"
format_stamp: "Formatted at 2026-08-10 01:47:06 on dist-test-slave-kvfs"
server_key: "8bfc8c537274497366aad092125cb4b9"
server_key_iv: "48db9c9d9e6aa99883b262139d1679cf"
server_key_version: "encryptionkey@0"
I20260810 01:47:06.665526 17648 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 01:47:06.669550 17666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:06.671887 17648 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 01:47:06.672139 17648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "4ca61933fb3a4dd89a0ff3c7ea581535"
format_stamp: "Formatted at 2026-08-10 01:47:06 on dist-test-slave-kvfs"
server_key: "8bfc8c537274497366aad092125cb4b9"
server_key_iv: "48db9c9d9e6aa99883b262139d1679cf"
server_key_version: "encryptionkey@0"
I20260810 01:47:06.672345 17648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:06.698295 17648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:06.699321 17648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:06.699520 17648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:06.708417 17648 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:42881
I20260810 01:47:06.708446 17747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:42881 every 8 connection(s)
I20260810 01:47:06.709724 17648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:47:06.714025 17750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:06.719539 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 17648
I20260810 01:47:06.719671 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:47:06.720009 9526 external_mini_cluster.cc:1428] Setting key a1d6a679585e63594c80fab838769e93
I20260810 01:47:06.721812 17750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ca61933fb3a4dd89a0ff3c7ea581535: Bootstrap starting.
I20260810 01:47:06.724864 17750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ca61933fb3a4dd89a0ff3c7ea581535: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:06.725838 17750 log.cc:824] T 00000000000000000000000000000000 P 4ca61933fb3a4dd89a0ff3c7ea581535: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:06.728324 17750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ca61933fb3a4dd89a0ff3c7ea581535: No bootstrap required, opened a new log
I20260810 01:47:06.731479 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:06.721919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56112 (local address 127.9.77.190:42881)
0810 01:47:06.722792 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:06.722823 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:06.722871 (+ 48us) server_negotiation.cc:407] Connection header received
0810 01:47:06.723678 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:06.723700 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:06.724305 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:06.724731 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:06.725293 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:06.726492 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:06.727521 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:06.728148 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:06.728338 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:06.729390 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:06.729395 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:06.729412 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:47:06.729900 (+ 488us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:06.729968 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:47:06.729990 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:06.730108 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:06.730256 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 01:47:06.730573 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":170,"threads_started":1}
I20260810 01:47:06.732726 17750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ca61933fb3a4dd89a0ff3c7ea581535 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca61933fb3a4dd89a0ff3c7ea581535" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 42881 } }
I20260810 01:47:06.733062 17750 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ca61933fb3a4dd89a0ff3c7ea581535 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:06.733124 17750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ca61933fb3a4dd89a0ff3c7ea581535 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ca61933fb3a4dd89a0ff3c7ea581535, State: Initialized, Role: FOLLOWER
I20260810 01:47:06.733680 17750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ca61933fb3a4dd89a0ff3c7ea581535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca61933fb3a4dd89a0ff3c7ea581535" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 42881 } }
I20260810 01:47:06.733899 17750 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ca61933fb3a4dd89a0ff3c7ea581535 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:47:06.734001 17750 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ca61933fb3a4dd89a0ff3c7ea581535 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:47:06.734117 17750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ca61933fb3a4dd89a0ff3c7ea581535 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:06.734974 9526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 01:47:06.735519 17750 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ca61933fb3a4dd89a0ff3c7ea581535 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca61933fb3a4dd89a0ff3c7ea581535" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 42881 } }
I20260810 01:47:06.735940 17750 leader_election.cc:302] T 00000000000000000000000000000000 P 4ca61933fb3a4dd89a0ff3c7ea581535 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ca61933fb3a4dd89a0ff3c7ea581535; no voters:
I20260810 01:47:06.736413 17750 leader_election.cc:288] T 00000000000000000000000000000000 P 4ca61933fb3a4dd89a0ff3c7ea581535 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:47:06.736567 17759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ca61933fb3a4dd89a0ff3c7ea581535 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:47:06.736820 17759 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ca61933fb3a4dd89a0ff3c7ea581535 [term 1 LEADER]: Becoming Leader. State: Replica: 4ca61933fb3a4dd89a0ff3c7ea581535, State: Running, Role: LEADER
I20260810 01:47:06.737231 17759 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ca61933fb3a4dd89a0ff3c7ea581535 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca61933fb3a4dd89a0ff3c7ea581535" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 42881 } }
I20260810 01:47:06.737989 17750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ca61933fb3a4dd89a0ff3c7ea581535 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:47:06.739403 17761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ca61933fb3a4dd89a0ff3c7ea581535 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ca61933fb3a4dd89a0ff3c7ea581535. Latest consensus state: current_term: 1 leader_uuid: "4ca61933fb3a4dd89a0ff3c7ea581535" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca61933fb3a4dd89a0ff3c7ea581535" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 42881 } } }
I20260810 01:47:06.739581 17761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ca61933fb3a4dd89a0ff3c7ea581535 [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:06.739964 17760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ca61933fb3a4dd89a0ff3c7ea581535 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ca61933fb3a4dd89a0ff3c7ea581535" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca61933fb3a4dd89a0ff3c7ea581535" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 42881 } } }
I20260810 01:47:06.740113 17760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ca61933fb3a4dd89a0ff3c7ea581535 [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:06.740298 17771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:47:06.744377 17771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:47:06.752664 17771 catalog_manager.cc:1347] Generated new cluster ID: 29600001288d4739938dbb53cdbb550c
I20260810 01:47:06.752796 17771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:47:06.769722 17771 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:47:06.770998 17771 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:47:06.776994 17771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ca61933fb3a4dd89a0ff3c7ea581535: Generated new TSK 0
I20260810 01:47:06.778616 17771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:47:06.805424 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:06.799540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56126 (local address 127.9.77.190:42881)
0810 01:47:06.799741 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:06.799747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:06.799937 (+ 190us) server_negotiation.cc:407] Connection header received
0810 01:47:06.800185 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:06.800187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:06.800250 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:06.800364 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0810 01:47:06.800678 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:06.801308 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:06.802622 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:06.802868 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:06.802952 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:06.803367 (+ 415us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 01:47:06.804928 (+ 1561us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41565/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 01:47:06.805249 (+ 321us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56126: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41565/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":65}
W20260810 01:47:06.805543 17754 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56126: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41565/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 01:47:06.808195 17792 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:06.799437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:42881 (local address 127.0.0.1:56126)
0810 01:47:06.799828 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:06.799845 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:06.799992 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:06.800390 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:06.800394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:06.800403 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0810 01:47:06.800558 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:06.800565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:06.801453 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:06.801457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:06.802495 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:06.802506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:06.803004 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:06.803008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:06.803105 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:06.803137 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 01:47:06.805116 (+ 1979us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41565/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 01:47:06.807937 (+ 2821us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.77.190:42881: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41565/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":2596,"thread_start_us":145,"threads_started":1}
I20260810 01:47:06.808931 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 17648
E20260810 01:47:06.837249 17793 webserver.cc:533] Webserver: sslize failed:
E20260810 01:47:06.837394 17793 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-10T01:47:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (398 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T01:47:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:47:06Z Disabled control of system clock
I20260810 01:47:06.887681 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:37961
--webserver_interface=127.9.77.190
--webserver_port=0
--builtin_ntp_servers=127.9.77.148:42497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:37961
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786326392877349-9526-0/testchainca.cert with env {}
W20260810 01:47:07.019943 17800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:07.020473 17800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:07.020531 17800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:07.024796 17800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:47:07.024854 17800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:07.024871 17800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:07.024888 17800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:47:07.024904 17800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 01:47:07.024920 17800 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 01:47:07.024933 17800 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 01:47:07.030573 17800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:42497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:37961
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:37961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786326392877349-9526-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:07.031834 17800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:07.033160 17800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:07.043608 17812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:07.043630 17806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:07.043649 17808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:07.044092 17800 server_base.cc:1034] running on GCE node
I20260810 01:47:07.044899 17800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:07.046298 17800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:07.047488 17800 hybrid_clock.cc:648] HybridClock initialized: now 1786326427047455 us; error 50 us; skew 500 ppm
I20260810 01:47:07.049980 17800 webserver.cc:460] Webserver started at http://127.9.77.190:35783/ using document root <none> and password file <none>
I20260810 01:47:07.050750 17800 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:07.050836 17800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:07.051112 17800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:07.053846 17800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "179f5428bdc441919200839bc4bd63a9"
format_stamp: "Formatted at 2026-08-10 01:47:07 on dist-test-slave-kvfs"
server_key: "7a7ed205db631e09690467c83cc22029"
server_key_iv: "ff114223f08a2460307e5ee503b1a66d"
server_key_version: "encryptionkey@0"
I20260810 01:47:07.054571 17800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "179f5428bdc441919200839bc4bd63a9"
format_stamp: "Formatted at 2026-08-10 01:47:07 on dist-test-slave-kvfs"
server_key: "7a7ed205db631e09690467c83cc22029"
server_key_iv: "ff114223f08a2460307e5ee503b1a66d"
server_key_version: "encryptionkey@0"
I20260810 01:47:07.059401 17800 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 01:47:07.062556 17820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:07.064316 17800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:47:07.064463 17800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "179f5428bdc441919200839bc4bd63a9"
format_stamp: "Formatted at 2026-08-10 01:47:07 on dist-test-slave-kvfs"
server_key: "7a7ed205db631e09690467c83cc22029"
server_key_iv: "ff114223f08a2460307e5ee503b1a66d"
server_key_version: "encryptionkey@0"
I20260810 01:47:07.064595 17800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:07.080446 17800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:07.081326 17800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:07.081557 17800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:07.090273 17800 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:37961
I20260810 01:47:07.090304 17901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:37961 every 8 connection(s)
I20260810 01:47:07.091519 17800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:47:07.095125 17902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:07.095566 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 17800
I20260810 01:47:07.095722 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:47:07.096159 9526 external_mini_cluster.cc:1428] Setting key 5054f82ff1493423432e4de216e80a03
I20260810 01:47:07.102450 17902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 179f5428bdc441919200839bc4bd63a9: Bootstrap starting.
I20260810 01:47:07.105677 17902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 179f5428bdc441919200839bc4bd63a9: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:07.106660 17902 log.cc:824] T 00000000000000000000000000000000 P 179f5428bdc441919200839bc4bd63a9: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:07.107270 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:07.098044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32980 (local address 127.9.77.190:37961)
0810 01:47:07.098883 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:07.098904 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:07.098950 (+ 46us) server_negotiation.cc:407] Connection header received
0810 01:47:07.099737 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:07.099757 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:07.100316 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:07.100658 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:07.101083 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:07.102226 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:07.103500 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:07.104013 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:07.104224 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:07.105178 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:07.105183 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:07.105198 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:47:07.105642 (+ 444us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:07.105705 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:47:07.105727 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:07.105866 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:07.106028 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 01:47:07.106322 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":179,"threads_started":1}
I20260810 01:47:07.109053 17902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 179f5428bdc441919200839bc4bd63a9: No bootstrap required, opened a new log
I20260810 01:47:07.110549 9526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 01:47:07.113502 17902 raft_consensus.cc:348] T 00000000000000000000000000000000 P 179f5428bdc441919200839bc4bd63a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179f5428bdc441919200839bc4bd63a9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37961 } }
I20260810 01:47:07.113803 17902 raft_consensus.cc:374] T 00000000000000000000000000000000 P 179f5428bdc441919200839bc4bd63a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:07.113888 17902 raft_consensus.cc:729] T 00000000000000000000000000000000 P 179f5428bdc441919200839bc4bd63a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 179f5428bdc441919200839bc4bd63a9, State: Initialized, Role: FOLLOWER
I20260810 01:47:07.114388 17902 consensus_queue.cc:260] T 00000000000000000000000000000000 P 179f5428bdc441919200839bc4bd63a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179f5428bdc441919200839bc4bd63a9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37961 } }
I20260810 01:47:07.114557 17902 raft_consensus.cc:388] T 00000000000000000000000000000000 P 179f5428bdc441919200839bc4bd63a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:47:07.114650 17902 raft_consensus.cc:482] T 00000000000000000000000000000000 P 179f5428bdc441919200839bc4bd63a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:47:07.114794 17902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 179f5428bdc441919200839bc4bd63a9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:07.115859 17902 raft_consensus.cc:504] T 00000000000000000000000000000000 P 179f5428bdc441919200839bc4bd63a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179f5428bdc441919200839bc4bd63a9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37961 } }
I20260810 01:47:07.116305 17902 leader_election.cc:302] T 00000000000000000000000000000000 P 179f5428bdc441919200839bc4bd63a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 179f5428bdc441919200839bc4bd63a9; no voters:
I20260810 01:47:07.116701 17902 leader_election.cc:288] T 00000000000000000000000000000000 P 179f5428bdc441919200839bc4bd63a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:47:07.116859 17914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 179f5428bdc441919200839bc4bd63a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:47:07.117270 17914 raft_consensus.cc:686] T 00000000000000000000000000000000 P 179f5428bdc441919200839bc4bd63a9 [term 1 LEADER]: Becoming Leader. State: Replica: 179f5428bdc441919200839bc4bd63a9, State: Running, Role: LEADER
I20260810 01:47:07.117624 17914 consensus_queue.cc:237] T 00000000000000000000000000000000 P 179f5428bdc441919200839bc4bd63a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179f5428bdc441919200839bc4bd63a9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37961 } }
I20260810 01:47:07.118296 17902 sys_catalog.cc:564] T 00000000000000000000000000000000 P 179f5428bdc441919200839bc4bd63a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:47:07.119823 17916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 179f5428bdc441919200839bc4bd63a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "179f5428bdc441919200839bc4bd63a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179f5428bdc441919200839bc4bd63a9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37961 } } }
I20260810 01:47:07.119982 17916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 179f5428bdc441919200839bc4bd63a9 [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:07.120193 17917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 179f5428bdc441919200839bc4bd63a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 179f5428bdc441919200839bc4bd63a9. Latest consensus state: current_term: 1 leader_uuid: "179f5428bdc441919200839bc4bd63a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179f5428bdc441919200839bc4bd63a9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37961 } } }
I20260810 01:47:07.120402 17917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 179f5428bdc441919200839bc4bd63a9 [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:07.120537 17928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:47:07.124120 17928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:47:07.130417 17928 catalog_manager.cc:1347] Generated new cluster ID: 4b4c711039a0442d84f816753b841e16
I20260810 01:47:07.130518 17928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:47:07.135898 17928 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:47:07.136934 17928 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:47:07.141901 17928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 179f5428bdc441919200839bc4bd63a9: Generated new TSK 0
I20260810 01:47:07.142675 17928 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:47:07.596010 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:07.155033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32984 (local address 127.9.77.190:37961)
0810 01:47:07.155193 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:07.155198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:07.155438 (+ 240us) server_negotiation.cc:407] Connection header received
0810 01:47:07.155657 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:07.155660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:07.155721 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:07.155838 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0810 01:47:07.156135 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:07.156731 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:07.158055 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:07.158327 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:07.158419 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:07.158776 (+ 357us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 01:47:07.595458 (+436682us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 01:47:07.595843 (+ 385us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32984: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":59}
W20260810 01:47:07.596192 17906 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32984: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 01:47:07.596462 17967 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:07.154968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37961 (local address 127.0.0.1:32984)
0810 01:47:07.155322 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:07.155338 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:07.155479 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:07.155857 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:07.155861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:07.155870 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0810 01:47:07.156008 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:07.156015 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:07.156862 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:07.156866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:07.157919 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:07.157930 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:07.158455 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:07.158459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:07.158558 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:07.158591 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 01:47:07.595810 (+437219us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 01:47:07.596145 (+ 335us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.77.190:37961: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":140,"threads_started":1}
I20260810 01:47:07.597213 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 17800
2026-08-10T01:47:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (755 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (170 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (150 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:47:07 dist-test-slave-kvfs krb5kdc[17979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:47:07 dist-test-slave-kvfs krb5kdc[17979](info): set up 2 sockets
Aug 10 01:47:07 dist-test-slave-kvfs krb5kdc[17979](info): commencing operation
krb5kdc: starting...
W20260810 01:47:09.264623 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.314s user 0.003s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:47:09 dist-test-slave-kvfs krb5kdc[17979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:47:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:47:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:47:09.412678 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:34749
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:43773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:34749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:09.549677 17996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:09.550081 17996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:09.550156 17996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:09.554308 17996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:47:09.554411 17996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:09.554447 17996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:09.554484 17996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:47:09.554522 17996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:47:09.560146 17996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:43773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:34749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:34749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:09.561566 17996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:09.562883 17996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:09.573220 18007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:09.573252 18006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:09.573477 18009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:09.573405 17996 server_base.cc:1034] running on GCE node
I20260810 01:47:09.574329 17996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:09.575685 17996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:09.576897 17996 hybrid_clock.cc:648] HybridClock initialized: now 1786326429576859 us; error 52 us; skew 500 ppm
Aug 10 01:47:09 dist-test-slave-kvfs krb5kdc[17979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326429, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:09.580299 17996 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:47:09.581485 17996 webserver.cc:460] Webserver started at http://127.9.77.190:40045/ using document root <none> and password file <none>
I20260810 01:47:09.582243 17996 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:09.582348 17996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:09.582644 17996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:09.585325 17996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "e8c7f32e57a14b2baa0d7ad574228ff3"
format_stamp: "Formatted at 2026-08-10 01:47:09 on dist-test-slave-kvfs"
server_key: "ac48afa860cf9569a40878336f1faece"
server_key_iv: "65f547b60a1304b9ad5fa62a7336a2a3"
server_key_version: "encryptionkey@0"
I20260810 01:47:09.586004 17996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "e8c7f32e57a14b2baa0d7ad574228ff3"
format_stamp: "Formatted at 2026-08-10 01:47:09 on dist-test-slave-kvfs"
server_key: "ac48afa860cf9569a40878336f1faece"
server_key_iv: "65f547b60a1304b9ad5fa62a7336a2a3"
server_key_version: "encryptionkey@0"
I20260810 01:47:09.590544 17996 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 01:47:09.593612 18025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:09.595160 17996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 01:47:09.595315 17996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "e8c7f32e57a14b2baa0d7ad574228ff3"
format_stamp: "Formatted at 2026-08-10 01:47:09 on dist-test-slave-kvfs"
server_key: "ac48afa860cf9569a40878336f1faece"
server_key_iv: "65f547b60a1304b9ad5fa62a7336a2a3"
server_key_version: "encryptionkey@0"
I20260810 01:47:09.595449 17996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:09.608645 17996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:09.612391 17996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:09.612625 17996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:09.620821 17996 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:34749
I20260810 01:47:09.620888 18120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:34749 every 8 connection(s)
I20260810 01:47:09.622109 17996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:47:09.625905 18123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:09.632236 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 17996
I20260810 01:47:09.632426 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:47:09.632782 9526 external_mini_cluster.cc:1428] Setting key 866285824ae5bf438e225219453584e4
I20260810 01:47:09.633133 18123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8c7f32e57a14b2baa0d7ad574228ff3: Bootstrap starting.
I20260810 01:47:09.636420 18123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8c7f32e57a14b2baa0d7ad574228ff3: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:09.637612 18123 log.cc:824] T 00000000000000000000000000000000 P e8c7f32e57a14b2baa0d7ad574228ff3: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:09.640362 18123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8c7f32e57a14b2baa0d7ad574228ff3: No bootstrap required, opened a new log
Aug 10 01:47:09 dist-test-slave-kvfs krb5kdc[17979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:47:09.645982 18123 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8c7f32e57a14b2baa0d7ad574228ff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c7f32e57a14b2baa0d7ad574228ff3" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 34749 } }
I20260810 01:47:09.646377 18123 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8c7f32e57a14b2baa0d7ad574228ff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:09.646446 18123 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8c7f32e57a14b2baa0d7ad574228ff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8c7f32e57a14b2baa0d7ad574228ff3, State: Initialized, Role: FOLLOWER
I20260810 01:47:09.647104 18123 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8c7f32e57a14b2baa0d7ad574228ff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c7f32e57a14b2baa0d7ad574228ff3" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 34749 } }
I20260810 01:47:09.647315 18123 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8c7f32e57a14b2baa0d7ad574228ff3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:47:09.647420 18123 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8c7f32e57a14b2baa0d7ad574228ff3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:47:09.647583 18123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8c7f32e57a14b2baa0d7ad574228ff3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:09.648819 18123 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8c7f32e57a14b2baa0d7ad574228ff3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c7f32e57a14b2baa0d7ad574228ff3" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 34749 } }
I20260810 01:47:09.649263 18123 leader_election.cc:302] T 00000000000000000000000000000000 P e8c7f32e57a14b2baa0d7ad574228ff3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8c7f32e57a14b2baa0d7ad574228ff3; no voters:
I20260810 01:47:09.649703 18123 leader_election.cc:288] T 00000000000000000000000000000000 P e8c7f32e57a14b2baa0d7ad574228ff3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:47:09.649797 18133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8c7f32e57a14b2baa0d7ad574228ff3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:47:09.650064 18133 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8c7f32e57a14b2baa0d7ad574228ff3 [term 1 LEADER]: Becoming Leader. State: Replica: e8c7f32e57a14b2baa0d7ad574228ff3, State: Running, Role: LEADER
I20260810 01:47:09.650444 18133 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8c7f32e57a14b2baa0d7ad574228ff3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c7f32e57a14b2baa0d7ad574228ff3" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 34749 } }
I20260810 01:47:09.650950 18131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:09.634500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55370 (local address 127.9.77.190:34749)
0810 01:47:09.635025 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:09.635046 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:09.635094 (+ 48us) server_negotiation.cc:407] Connection header received
0810 01:47:09.635903 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:09.635924 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:09.636457 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:09.636894 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:09.638018 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:09.639227 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:09.640503 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:09.641109 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:09.641252 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:09.644710 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:09.644719 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:09.644747 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:09.644786 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:09.646992 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:09.647589 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:09.647596 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:09.647606 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:09.647799 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:09.648142 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:09.648146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:09.648148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:09.648661 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:09.648894 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:09.649240 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 01:47:09.649668 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":176,"threads_started":1}
I20260810 01:47:09.651392 18123 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8c7f32e57a14b2baa0d7ad574228ff3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:47:09.652443 18135 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c7f32e57a14b2baa0d7ad574228ff3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8c7f32e57a14b2baa0d7ad574228ff3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c7f32e57a14b2baa0d7ad574228ff3" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 34749 } } }
I20260810 01:47:09.652617 18135 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c7f32e57a14b2baa0d7ad574228ff3 [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:09.652998 18134 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c7f32e57a14b2baa0d7ad574228ff3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8c7f32e57a14b2baa0d7ad574228ff3. Latest consensus state: current_term: 1 leader_uuid: "e8c7f32e57a14b2baa0d7ad574228ff3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c7f32e57a14b2baa0d7ad574228ff3" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 34749 } } }
I20260810 01:47:09.653107 18134 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c7f32e57a14b2baa0d7ad574228ff3 [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:09.654340 18145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:47:09.658003 18145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:47:09.665123 18145 catalog_manager.cc:1347] Generated new cluster ID: 2b0040aa4d494c2b96adb87ac7e2242a
I20260810 01:47:09.665221 18145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:47:09.675403 18145 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:47:09.676868 18145 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:47:09.688053 18145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8c7f32e57a14b2baa0d7ad574228ff3: Generated new TSK 0
I20260810 01:47:09.689180 18145 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.77.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
I20260810 01:47:09.732545 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:0
--local_ip_for_outbound_sockets=127.9.77.129
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:34749
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:43773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:09.873934 18160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:09.874297 18160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:09.874347 18160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:09.878664 18160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:09.878743 18160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:09.878808 18160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:47:09.884732 18160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:43773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:34749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:09.886211 18160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:09.887616 18160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:09.899693 18174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:09.899725 18167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:09.899693 18165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:09.900059 18160 server_base.cc:1034] running on GCE node
I20260810 01:47:09.900704 18160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:09.901563 18160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:09.902746 18160 hybrid_clock.cc:648] HybridClock initialized: now 1786326429902729 us; error 31 us; skew 500 ppm
Aug 10 01:47:09 dist-test-slave-kvfs krb5kdc[17979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326429, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:09.906432 18160 init.cc:377] Logged in from keytab as kudu/127.9.77.129@KRBTEST.COM (short username kudu)
I20260810 01:47:09.907639 18160 webserver.cc:460] Webserver started at http://127.9.77.129:40125/ using document root <none> and password file <none>
I20260810 01:47:09.908500 18160 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:09.908578 18160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:09.908872 18160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:09.912006 18160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-0/data/instance:
uuid: "b635a2f013f24782a97f47b6ff0e5fef"
format_stamp: "Formatted at 2026-08-10 01:47:09 on dist-test-slave-kvfs"
server_key: "35fad5d372f4ff7944880879a4add8f5"
server_key_iv: "873d58eced6d9addde21bce935277fdc"
server_key_version: "encryptionkey@0"
I20260810 01:47:09.912847 18160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance:
uuid: "b635a2f013f24782a97f47b6ff0e5fef"
format_stamp: "Formatted at 2026-08-10 01:47:09 on dist-test-slave-kvfs"
server_key: "35fad5d372f4ff7944880879a4add8f5"
server_key_iv: "873d58eced6d9addde21bce935277fdc"
server_key_version: "encryptionkey@0"
I20260810 01:47:09.918059 18160 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260810 01:47:09.921440 18183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:09.923127 18160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:47:09.923317 18160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "b635a2f013f24782a97f47b6ff0e5fef"
format_stamp: "Formatted at 2026-08-10 01:47:09 on dist-test-slave-kvfs"
server_key: "35fad5d372f4ff7944880879a4add8f5"
server_key_iv: "873d58eced6d9addde21bce935277fdc"
server_key_version: "encryptionkey@0"
I20260810 01:47:09.923501 18160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:09.959441 18160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:09.963665 18160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:09.963946 18160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:09.964810 18160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:09.965903 18160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:09.965986 18160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:09.966085 18160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:09.966133 18160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:09.977406 18160 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:43019
I20260810 01:47:09.977432 18364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:43019 every 8 connection(s)
I20260810 01:47:09.978596 18160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:47:09.984326 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 18160
I20260810 01:47:09.984506 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance
I20260810 01:47:09.984889 9526 external_mini_cluster.cc:1428] Setting key 1fd0fff958ded5536ea222538e87f2df
Aug 10 01:47:09 dist-test-slave-kvfs krb5kdc[17979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326429, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:47:09.996542 18131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:09.981306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:53285 (local address 127.9.77.190:34749)
0810 01:47:09.981505 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:09.981510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:09.982750 (+ 1240us) server_negotiation.cc:407] Connection header received
0810 01:47:09.984648 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:09.984652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:09.984738 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:09.984851 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:09.986576 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:09.987165 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:09.988579 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:09.988845 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:09.988977 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:09.992354 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:09.992357 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:09.992359 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:09.992388 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:09.994048 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:09.994644 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:09.994648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:09.994649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:09.994734 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:09.995145 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:09.995149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:09.995150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:09.995369 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:09.995491 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:09.996160 (+ 669us) server_negotiation.cc:299] Negotiation successful
0810 01:47:09.996327 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 01:47:09.997625 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:09.981811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:34749 (local address 127.9.77.129:53285)
0810 01:47:09.982452 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:09.982507 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:09.984311 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:09.985105 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:09.985114 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:09.985736 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:09.986405 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:09.986421 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:09.987311 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:09.987320 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:09.988443 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:09.988453 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:09.989026 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:09.989030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:09.989249 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:09.990038 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:09.990069 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:09.992149 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:09.994191 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:09.994198 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:09.994219 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:09.994518 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:09.994832 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:09.994836 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:09.994838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:09.995013 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:09.995508 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:09.995517 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:09.995836 (+ 319us) client_negotiation.cc:769] Sending connection context
0810 01:47:09.996102 (+ 266us) client_negotiation.cc:240] Negotiation successful
0810 01:47:09.996523 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":146,"threads_started":1}
WARNING: no policy specified for kudu/127.9.77.130@KRBTEST.COM; defaulting to no policy
I20260810 01:47:09.999269 18365 heartbeater.cc:347] Connected to a master server at 127.9.77.190:34749
I20260810 01:47:09.999616 18365 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:10.000419 18365 heartbeater.cc:511] Master 127.9.77.190:34749 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.130@KRBTEST.COM" created.
I20260810 01:47:10.002691 18060 ts_manager.cc:194] Registered new tserver with Master: b635a2f013f24782a97f47b6ff0e5fef (127.9.77.129:43019)
I20260810 01:47:10.004251 18060 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.129@KRBTEST.COM'} at 127.9.77.129:53285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
I20260810 01:47:10.024883 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:0
--local_ip_for_outbound_sockets=127.9.77.130
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:34749
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:43773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:10.161180 18371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:10.161522 18371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:10.161571 18371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:10.166369 18371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:10.166471 18371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:10.166570 18371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:47:10.173221 18371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:43773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:34749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:10.174721 18371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:10.176124 18371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:10.188411 18384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:10.188422 18382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:10.188411 18380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:10.188896 18371 server_base.cc:1034] running on GCE node
I20260810 01:47:10.189436 18371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:10.190263 18371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:10.191499 18371 hybrid_clock.cc:648] HybridClock initialized: now 1786326430191479 us; error 32 us; skew 500 ppm
Aug 10 01:47:10 dist-test-slave-kvfs krb5kdc[17979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326430, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:10.194849 18371 init.cc:377] Logged in from keytab as kudu/127.9.77.130@KRBTEST.COM (short username kudu)
I20260810 01:47:10.195890 18371 webserver.cc:460] Webserver started at http://127.9.77.130:37939/ using document root <none> and password file <none>
I20260810 01:47:10.196633 18371 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:10.196702 18371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:10.196916 18371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:10.199427 18371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-1/data/instance:
uuid: "73b7092099df49e9bf83b368a371c869"
format_stamp: "Formatted at 2026-08-10 01:47:10 on dist-test-slave-kvfs"
server_key: "b56ad650bbc7057c2096e44fa38e833b"
server_key_iv: "1839887afc9d050dbdd2fdf2457752b4"
server_key_version: "encryptionkey@0"
I20260810 01:47:10.200115 18371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance:
uuid: "73b7092099df49e9bf83b368a371c869"
format_stamp: "Formatted at 2026-08-10 01:47:10 on dist-test-slave-kvfs"
server_key: "b56ad650bbc7057c2096e44fa38e833b"
server_key_iv: "1839887afc9d050dbdd2fdf2457752b4"
server_key_version: "encryptionkey@0"
I20260810 01:47:10.204778 18371 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 01:47:10.207774 18398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:10.209479 18371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:47:10.209641 18371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "73b7092099df49e9bf83b368a371c869"
format_stamp: "Formatted at 2026-08-10 01:47:10 on dist-test-slave-kvfs"
server_key: "b56ad650bbc7057c2096e44fa38e833b"
server_key_iv: "1839887afc9d050dbdd2fdf2457752b4"
server_key_version: "encryptionkey@0"
I20260810 01:47:10.209785 18371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:10.243610 18371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:10.247871 18371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:10.248180 18371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:10.249010 18371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:10.250099 18371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:10.250151 18371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:10.250228 18371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:10.250262 18371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:10.262001 18371 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:41599
I20260810 01:47:10.262022 18581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:41599 every 8 connection(s)
I20260810 01:47:10.263185 18371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:47:10.266217 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 18371
I20260810 01:47:10.266388 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance
I20260810 01:47:10.266779 9526 external_mini_cluster.cc:1428] Setting key 9f40fc7a91ed2f560abcce6589a4a911
Aug 10 01:47:10 dist-test-slave-kvfs krb5kdc[17979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326430, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.77.131@KRBTEST.COM; defaulting to no policy
I20260810 01:47:10.282967 18131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:10.265849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:50627 (local address 127.9.77.190:34749)
0810 01:47:10.266084 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:10.266089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:10.267376 (+ 1287us) server_negotiation.cc:407] Connection header received
0810 01:47:10.269517 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:10.269524 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:10.269603 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:10.269713 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:10.271253 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:10.272214 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:10.274143 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:10.274412 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:10.274517 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:10.278352 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:10.278355 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:10.278358 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:10.278390 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:10.280292 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:10.280965 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:10.280968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:10.280969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:10.281059 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:10.281548 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:10.281551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:10.281552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:10.281809 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:10.281901 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:10.282618 (+ 717us) server_negotiation.cc:299] Negotiation successful
0810 01:47:10.282781 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 01:47:10.284160 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:10.266436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:34749 (local address 127.9.77.130:50627)
0810 01:47:10.267134 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:10.267217 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:10.269227 (+ 2010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:10.269959 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:10.269968 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:10.270435 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:10.271056 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:10.271073 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:10.272379 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:10.272386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:10.273995 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:10.274010 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:10.274595 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:10.274603 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:10.274866 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:10.275701 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:10.275733 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:10.278102 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:10.280432 (+ 2330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:10.280443 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:10.280469 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:10.280848 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:10.281193 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:10.281196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:10.281198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:10.281422 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:10.281913 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:10.281919 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:10.282229 (+ 310us) client_negotiation.cc:769] Sending connection context
0810 01:47:10.282485 (+ 256us) client_negotiation.cc:240] Negotiation successful
0810 01:47:10.282845 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":200,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.131@KRBTEST.COM" created.
I20260810 01:47:10.285705 18582 heartbeater.cc:347] Connected to a master server at 127.9.77.190:34749
I20260810 01:47:10.286044 18582 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:10.286813 18582 heartbeater.cc:511] Master 127.9.77.190:34749 requested a full tablet report, sending...
I20260810 01:47:10.288345 18060 ts_manager.cc:194] Registered new tserver with Master: 73b7092099df49e9bf83b368a371c869 (127.9.77.130:41599)
I20260810 01:47:10.289064 18060 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.130@KRBTEST.COM'} at 127.9.77.130:50627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
I20260810 01:47:10.307281 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:0
--local_ip_for_outbound_sockets=127.9.77.131
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:34749
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:43773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:10.445259 18589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:10.445672 18589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:10.445720 18589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:10.450250 18589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:10.450356 18589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:10.450466 18589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:47:10.456887 18589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:43773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:34749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:10.458395 18589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:10.459695 18589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:10.472594 18597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:10.472592 18598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:10.472987 18589 server_base.cc:1034] running on GCE node
W20260810 01:47:10.472592 18600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:10.473692 18589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:10.474488 18589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:10.475687 18589 hybrid_clock.cc:648] HybridClock initialized: now 1786326430475632 us; error 63 us; skew 500 ppm
Aug 10 01:47:10 dist-test-slave-kvfs krb5kdc[17979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326430, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:10.479365 18589 init.cc:377] Logged in from keytab as kudu/127.9.77.131@KRBTEST.COM (short username kudu)
I20260810 01:47:10.480628 18589 webserver.cc:460] Webserver started at http://127.9.77.131:37611/ using document root <none> and password file <none>
I20260810 01:47:10.481395 18589 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:10.481465 18589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:10.481783 18589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:10.484650 18589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-2/data/instance:
uuid: "2733079d27594b4da740ff7190180860"
format_stamp: "Formatted at 2026-08-10 01:47:10 on dist-test-slave-kvfs"
server_key: "6b95842ce88f5b46cf10c740e3f92038"
server_key_iv: "9dd4d7877c0b0bf85d42863f13f5c757"
server_key_version: "encryptionkey@0"
I20260810 01:47:10.485368 18589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance:
uuid: "2733079d27594b4da740ff7190180860"
format_stamp: "Formatted at 2026-08-10 01:47:10 on dist-test-slave-kvfs"
server_key: "6b95842ce88f5b46cf10c740e3f92038"
server_key_iv: "9dd4d7877c0b0bf85d42863f13f5c757"
server_key_version: "encryptionkey@0"
I20260810 01:47:10.490214 18589 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 01:47:10.493336 18609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:10.494876 18589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:47:10.495042 18589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "2733079d27594b4da740ff7190180860"
format_stamp: "Formatted at 2026-08-10 01:47:10 on dist-test-slave-kvfs"
server_key: "6b95842ce88f5b46cf10c740e3f92038"
server_key_iv: "9dd4d7877c0b0bf85d42863f13f5c757"
server_key_version: "encryptionkey@0"
I20260810 01:47:10.495183 18589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:10.517827 18589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:10.521920 18589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:10.522172 18589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:10.522958 18589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:10.524072 18589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:10.524152 18589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:10.524232 18589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:10.524274 18589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:10.535343 18589 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:46137
I20260810 01:47:10.535373 18787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:46137 every 8 connection(s)
I20260810 01:47:10.536558 18589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:47:10.537961 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 18589
I20260810 01:47:10.538077 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance
I20260810 01:47:10.538447 9526 external_mini_cluster.cc:1428] Setting key 41bfae06c2a5716ce53aed6ac9d30a12
Aug 10 01:47:10 dist-test-slave-kvfs krb5kdc[17979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326430, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:47:10.553543 18131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:10.538997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:46171 (local address 127.9.77.190:34749)
0810 01:47:10.539185 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:10.539189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:10.540408 (+ 1219us) server_negotiation.cc:407] Connection header received
0810 01:47:10.542024 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:10.542027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:10.542083 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:10.542154 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:10.543586 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:10.544272 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:10.545544 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:10.545778 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:10.545868 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:10.548850 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:10.548854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:10.548857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:10.548889 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:10.550558 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:10.551169 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:10.551173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:10.551175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:10.551303 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:10.551797 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:10.551801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:10.551802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:10.552144 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:10.552258 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:10.553151 (+ 893us) server_negotiation.cc:299] Negotiation successful
0810 01:47:10.553313 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 01:47:10.554682 18791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:10.539591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:34749 (local address 127.9.77.131:46171)
0810 01:47:10.540225 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:10.540288 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:10.541804 (+ 1516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:10.542366 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:10.542374 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:10.542836 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:10.543444 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:10.543460 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:10.544396 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:10.544400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:10.545413 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:10.545423 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:10.545925 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:10.545928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:10.546091 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:10.546780 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:10.546798 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:10.548657 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:10.550684 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:10.550690 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:10.550707 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:10.551034 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:10.551473 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:10.551477 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:10.551478 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:10.551698 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:10.552260 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:10.552266 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:10.552719 (+ 453us) client_negotiation.cc:769] Sending connection context
0810 01:47:10.553007 (+ 288us) client_negotiation.cc:240] Negotiation successful
0810 01:47:10.553476 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":168,"threads_started":1}
I20260810 01:47:10.555956 18788 heartbeater.cc:347] Connected to a master server at 127.9.77.190:34749
I20260810 01:47:10.556267 18788 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:10.556870 18788 heartbeater.cc:511] Master 127.9.77.190:34749 requested a full tablet report, sending...
I20260810 01:47:10.558091 18060 ts_manager.cc:194] Registered new tserver with Master: 2733079d27594b4da740ff7190180860 (127.9.77.131:46137)
I20260810 01:47:10.558748 18060 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.131@KRBTEST.COM'} at 127.9.77.131:46171
I20260810 01:47:10.565320 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/test-admin.keytab.
Aug 10 01:47:10 dist-test-slave-kvfs krb5kdc[17979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:10.585826 9526 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 01:47:10.589633 18796 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:10.587728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47392 (local address 127.9.77.129:43019)
0810 01:47:10.588226 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:10.588234 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:10.588265 (+ 31us) server_negotiation.cc:407] Connection header received
0810 01:47:10.588332 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:10.588339 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:10.588594 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:10.588748 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:10.589381 (+ 633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47392: BlockingRecv error: recv got EOF from 127.0.0.1:47392 (error 108)
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":186,"threads_started":1}
W20260810 01:47:10.589792 9526 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/krb5cc' of type FILE
Aug 10 01:47:10 dist-test-slave-kvfs krb5kdc[17979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:10.591146 9526 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 01:47:10 dist-test-slave-kvfs krb5kdc[17979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:10.592648 9526 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 01:47:10 dist-test-slave-kvfs krb5kdc[17979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.129@KRBTEST.COM
I20260810 01:47:10.604882 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:10.593952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47404 (local address 127.9.77.129:43019)
0810 01:47:10.594115 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:10.594123 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:10.594144 (+ 21us) server_negotiation.cc:407] Connection header received
0810 01:47:10.594361 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:10.594364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:10.594423 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:10.594525 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:10.595237 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:10.596193 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:10.597162 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:10.597657 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:10.597794 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:10.600937 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:10.600951 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:10.600970 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:10.601001 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:10.602910 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:10.603383 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:10.603389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:10.603391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:10.603493 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:10.603746 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:10.603748 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:10.603750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:10.604169 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:10.604382 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:10.604566 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 01:47:10.604726 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260810 01:47:10.610049 18796 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:10.608728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47416 (local address 127.9.77.129:43019)
0810 01:47:10.608887 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:10.608898 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:10.608960 (+ 62us) server_negotiation.cc:407] Connection header received
0810 01:47:10.609083 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:10.609085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:10.609129 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:10.609199 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:10.609895 (+ 696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47416: BlockingRecv error: recv got EOF from 127.0.0.1:47416 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260810 01:47:10.610081 9526 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/krb5cc' of type FILE
Aug 10 01:47:10 dist-test-slave-kvfs krb5kdc[17979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:10.611451 9526 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 01:47:10 dist-test-slave-kvfs krb5kdc[17979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:10.612686 9526 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 01:47:10 dist-test-slave-kvfs krb5kdc[17979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.129@KRBTEST.COM
I20260810 01:47:10.623817 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:10.613801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47432 (local address 127.9.77.129:43019)
0810 01:47:10.613950 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:10.613953 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:10.614068 (+ 115us) server_negotiation.cc:407] Connection header received
0810 01:47:10.614181 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:10.614205 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:10.614276 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:10.614355 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:10.615097 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:10.615758 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:10.616912 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:10.617161 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:10.617254 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:10.620007 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:10.620011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:10.620013 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:10.620099 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:10.621962 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:10.622459 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:10.622463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:10.622465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:10.622562 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:10.622830 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:10.622833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:10.622834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:10.623086 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:10.623261 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:10.623491 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 01:47:10.623642 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260810 01:47:10.627243 18796 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:10.626077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47440 (local address 127.9.77.129:43019)
0810 01:47:10.626225 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:10.626229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:10.626385 (+ 156us) server_negotiation.cc:407] Connection header received
0810 01:47:10.626504 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:10.626506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:10.626549 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:10.626615 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:10.627092 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47440: BlockingRecv error: recv got EOF from 127.0.0.1:47440 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 01:47:10.627485 9526 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786326392877349-9526-0/krb5kdc/krb5cc' of type FILE
Aug 10 01:47:10 dist-test-slave-kvfs krb5kdc[17979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:10.628969 9526 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 01:47:10 dist-test-slave-kvfs krb5kdc[17979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:10.630017 9526 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 01:47:10 dist-test-slave-kvfs krb5kdc[17979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.129@KRBTEST.COM
I20260810 01:47:10.639901 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:10.631033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47454 (local address 127.9.77.129:43019)
0810 01:47:10.631155 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:10.631159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:10.631264 (+ 105us) server_negotiation.cc:407] Connection header received
0810 01:47:10.631422 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:10.631425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:10.631471 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:10.631570 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:10.632152 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:10.632720 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:10.633591 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:10.633837 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:10.633952 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:10.636563 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:10.636567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:10.636570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:10.636596 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:10.638092 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:10.638572 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:10.638576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:10.638578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:10.638669 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:10.638961 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:10.638965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:10.638967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:10.639169 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:10.639351 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:10.639580 (+ 229us) server_negotiation.cc:299] Negotiation successful
0810 01:47:10.639712 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 01:47:10.641220 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 18160
I20260810 01:47:10.670105 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 18371
I20260810 01:47:10.693694 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 18589
I20260810 01:47:10.708405 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 17996
2026-08-10T01:47:10Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2795 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:47:10 dist-test-slave-kvfs krb5kdc[18817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:47:10 dist-test-slave-kvfs krb5kdc[18817](info): set up 2 sockets
Aug 10 01:47:10 dist-test-slave-kvfs krb5kdc[18817](info): commencing operation
krb5kdc: starting...
W20260810 01:47:12.027936 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.281s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:47:12 dist-test-slave-kvfs krb5kdc[18817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:47:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:47:12Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.190.keytab.
Entry for principal oryx/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.190.keytab.
I20260810 01:47:12.180243 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:33329
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.190.keytab
--principal=oryx/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:33329
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:12.315889 18838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:12.316365 18838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:12.316462 18838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:12.320791 18838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:47:12.320917 18838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:12.320956 18838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:12.320995 18838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:47:12.321033 18838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:47:12.326995 18838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:33329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.190.keytab
--principal=oryx/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:33329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:12.328575 18838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:12.329952 18838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:12.341138 18851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:12.341151 18847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:12.341380 18846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:12.341513 18838 server_base.cc:1034] running on GCE node
I20260810 01:47:12.342519 18838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:12.344127 18838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:12.345371 18838 hybrid_clock.cc:648] HybridClock initialized: now 1786326432345346 us; error 50 us; skew 500 ppm
Aug 10 01:47:12 dist-test-slave-kvfs krb5kdc[18817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326432, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:12.351722 18838 init.cc:377] Logged in from keytab as oryx/127.9.77.190@KRBTEST.COM (short username oryx)
I20260810 01:47:12.352990 18838 webserver.cc:460] Webserver started at http://127.9.77.190:43181/ using document root <none> and password file <none>
I20260810 01:47:12.353731 18838 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:12.353835 18838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:12.354130 18838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:12.356832 18838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "9ae815b0781f4d19aba6d526311d18c9"
format_stamp: "Formatted at 2026-08-10 01:47:12 on dist-test-slave-kvfs"
server_key: "ff23ce6a510a742c653557044b1d0fc3"
server_key_iv: "4d43194a8d7e7aa10805c2552580c3f6"
server_key_version: "encryptionkey@0"
I20260810 01:47:12.357542 18838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "9ae815b0781f4d19aba6d526311d18c9"
format_stamp: "Formatted at 2026-08-10 01:47:12 on dist-test-slave-kvfs"
server_key: "ff23ce6a510a742c653557044b1d0fc3"
server_key_iv: "4d43194a8d7e7aa10805c2552580c3f6"
server_key_version: "encryptionkey@0"
I20260810 01:47:12.362345 18838 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 01:47:12.365387 18864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:12.366901 18838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:47:12.367074 18838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "9ae815b0781f4d19aba6d526311d18c9"
format_stamp: "Formatted at 2026-08-10 01:47:12 on dist-test-slave-kvfs"
server_key: "ff23ce6a510a742c653557044b1d0fc3"
server_key_iv: "4d43194a8d7e7aa10805c2552580c3f6"
server_key_version: "encryptionkey@0"
I20260810 01:47:12.367205 18838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:12.384613 18838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:12.388625 18838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:12.388892 18838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:12.397591 18838 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:33329
I20260810 01:47:12.397948 18945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:33329 every 8 connection(s)
I20260810 01:47:12.398947 18838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:47:12.400585 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 18838
I20260810 01:47:12.400739 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:47:12.401108 9526 external_mini_cluster.cc:1428] Setting key d509e4407b205e064f1f7d2e613725e9
I20260810 01:47:12.402911 18946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:12.411168 18946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9: Bootstrap starting.
Aug 10 01:47:12 dist-test-slave-kvfs krb5kdc[18817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM
I20260810 01:47:12.414481 18946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:12.415369 18946 log.cc:824] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:12.418471 18946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9: No bootstrap required, opened a new log
I20260810 01:47:12.419880 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:12.402909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36546 (local address 127.9.77.190:33329)
0810 01:47:12.403454 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:12.403476 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:12.403531 (+ 55us) server_negotiation.cc:407] Connection header received
0810 01:47:12.404556 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:12.404584 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:12.405093 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:12.405527 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:12.406409 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:12.408131 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:12.409086 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:12.409862 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:12.410038 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:12.413558 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:12.413564 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:12.413580 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:12.413615 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:12.416013 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:12.416623 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:12.416629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:12.416637 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:12.416808 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:12.417115 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:12.417118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:12.417120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:12.417767 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:12.418004 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:12.418285 (+ 281us) server_negotiation.cc:299] Negotiation successful
0810 01:47:12.418666 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":163,"threads_started":1}
I20260810 01:47:12.423579 18946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } }
I20260810 01:47:12.423866 18946 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:12.423919 18946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae815b0781f4d19aba6d526311d18c9, State: Initialized, Role: FOLLOWER
I20260810 01:47:12.424460 18946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } }
I20260810 01:47:12.424609 18946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:47:12.424661 18946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:47:12.424755 18946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:12.425850 18946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } }
I20260810 01:47:12.426222 18946 leader_election.cc:302] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ae815b0781f4d19aba6d526311d18c9; no voters:
I20260810 01:47:12.426666 18946 leader_election.cc:288] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:47:12.426800 18952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:47:12.427058 18952 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [term 1 LEADER]: Becoming Leader. State: Replica: 9ae815b0781f4d19aba6d526311d18c9, State: Running, Role: LEADER
I20260810 01:47:12.427506 18952 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } }
I20260810 01:47:12.428411 18946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:47:12.429715 18953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ae815b0781f4d19aba6d526311d18c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } }
I20260810 01:47:12.429723 18955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ae815b0781f4d19aba6d526311d18c9. Latest consensus state: current_term: 1 leader_uuid: "9ae815b0781f4d19aba6d526311d18c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } }
I20260810 01:47:12.429922 18955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:12.429924 18953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:12.431133 18966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:47:12.434967 18966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:47:12.442103 18966 catalog_manager.cc:1347] Generated new cluster ID: 30255652128446de977d487ec2e58d4b
I20260810 01:47:12.442219 18966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:47:12.461086 18966 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:47:12.462719 18966 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:47:12.473134 18966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9: Generated new TSK 0
I20260810 01:47:12.474251 18966 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.77.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.77.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.77.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.129.keytab.
Entry for principal oryx/127.9.77.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.129.keytab.
I20260810 01:47:12.513435 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:0
--local_ip_for_outbound_sockets=127.9.77.129
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:33329
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.129.keytab
--principal=oryx/127.9.77.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:12.654901 18984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:12.655243 18984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:12.655288 18984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:12.659437 18984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:12.659508 18984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:12.659569 18984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:47:12.665337 18984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.129.keytab
--principal=oryx/127.9.77.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:33329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:12.666854 18984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:12.668259 18984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:12.680385 18994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:12.680424 18991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:12.680663 18992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:12.680637 18984 server_base.cc:1034] running on GCE node
I20260810 01:47:12.681448 18984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:12.682255 18984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:12.683431 18984 hybrid_clock.cc:648] HybridClock initialized: now 1786326432683414 us; error 29 us; skew 500 ppm
Aug 10 01:47:12 dist-test-slave-kvfs krb5kdc[18817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326432, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:12.686959 18984 init.cc:377] Logged in from keytab as oryx/127.9.77.129@KRBTEST.COM (short username oryx)
I20260810 01:47:12.688180 18984 webserver.cc:460] Webserver started at http://127.9.77.129:37359/ using document root <none> and password file <none>
I20260810 01:47:12.688915 18984 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:12.689018 18984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:12.689317 18984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:12.691917 18984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/data/instance:
uuid: "3ad0891514f242b8a78735404fbbcc19"
format_stamp: "Formatted at 2026-08-10 01:47:12 on dist-test-slave-kvfs"
server_key: "2e6c284eeb9f59cd42a4e889877f837c"
server_key_iv: "a712316147ddbe69f2211ef1012932bc"
server_key_version: "encryptionkey@0"
I20260810 01:47:12.692673 18984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance:
uuid: "3ad0891514f242b8a78735404fbbcc19"
format_stamp: "Formatted at 2026-08-10 01:47:12 on dist-test-slave-kvfs"
server_key: "2e6c284eeb9f59cd42a4e889877f837c"
server_key_iv: "a712316147ddbe69f2211ef1012932bc"
server_key_version: "encryptionkey@0"
I20260810 01:47:12.697271 18984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 01:47:12.700461 19003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:12.702086 18984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 01:47:12.702243 18984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "3ad0891514f242b8a78735404fbbcc19"
format_stamp: "Formatted at 2026-08-10 01:47:12 on dist-test-slave-kvfs"
server_key: "2e6c284eeb9f59cd42a4e889877f837c"
server_key_iv: "a712316147ddbe69f2211ef1012932bc"
server_key_version: "encryptionkey@0"
I20260810 01:47:12.702386 18984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:12.717814 18984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:12.721860 18984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:12.722107 18984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:12.722924 18984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:12.723970 18984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:12.724077 18984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:12.724172 18984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:12.724221 18984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:12.735543 18984 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:45485
I20260810 01:47:12.735641 19175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:45485 every 8 connection(s)
I20260810 01:47:12.736824 18984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:47:12.742981 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 18984
I20260810 01:47:12.743158 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance
I20260810 01:47:12.743553 9526 external_mini_cluster.cc:1428] Setting key 04460264c1b573e7688ec2a3ad55a956
Aug 10 01:47:12 dist-test-slave-kvfs krb5kdc[18817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326432, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM
I20260810 01:47:12.756783 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:12.739373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:51129 (local address 127.9.77.190:33329)
0810 01:47:12.739603 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:12.739607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:12.740848 (+ 1241us) server_negotiation.cc:407] Connection header received
0810 01:47:12.742645 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:12.742649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:12.742726 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:12.742862 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:12.744717 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:12.745511 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:12.747185 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:12.747654 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:12.747794 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:12.751471 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:12.751478 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:12.751483 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:12.751525 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:12.753723 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:12.754357 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:12.754362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:12.754364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:12.754495 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:12.755035 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:12.755042 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:12.755046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:12.755409 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:12.755550 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:12.756275 (+ 725us) server_negotiation.cc:299] Negotiation successful
0810 01:47:12.756488 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260810 01:47:12.757761 19178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:12.739897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.9.77.129:51129)
0810 01:47:12.740551 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:12.740607 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:12.742355 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:12.743129 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:12.743139 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:12.743728 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:12.744548 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:12.744568 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:12.745664 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:12.745667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:12.747032 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:12.747047 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:12.747844 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:12.747850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:12.748157 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:12.749044 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:12.749076 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:12.751258 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:12.753879 (+ 2621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:12.753885 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:12.753902 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:12.754219 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:12.754632 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:12.754636 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:12.754638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:12.754870 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:12.755557 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:12.755564 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:12.755906 (+ 342us) client_negotiation.cc:769] Sending connection context
0810 01:47:12.756163 (+ 257us) client_negotiation.cc:240] Negotiation successful
0810 01:47:12.756558 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":134,"threads_started":1}
WARNING: no policy specified for oryx/127.9.77.130@KRBTEST.COM; defaulting to no policy
I20260810 01:47:12.759498 19176 heartbeater.cc:347] Connected to a master server at 127.9.77.190:33329
I20260810 01:47:12.759824 19176 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:12.760526 19176 heartbeater.cc:511] Master 127.9.77.190:33329 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.77.130@KRBTEST.COM" created.
I20260810 01:47:12.762722 18891 ts_manager.cc:194] Registered new tserver with Master: 3ad0891514f242b8a78735404fbbcc19 (127.9.77.129:45485)
I20260810 01:47:12.764308 18891 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.77.129@KRBTEST.COM'} at 127.9.77.129:51129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.77.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.130.keytab.
Entry for principal oryx/127.9.77.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.130.keytab.
I20260810 01:47:12.784200 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:0
--local_ip_for_outbound_sockets=127.9.77.130
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:33329
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.130.keytab
--principal=oryx/127.9.77.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:12.919373 19181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:12.919732 19181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:12.919782 19181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:12.924149 19181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:12.924229 19181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:12.924324 19181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:47:12.930609 19181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.130.keytab
--principal=oryx/127.9.77.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:33329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:12.932070 19181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:12.933449 19181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:12.945272 19191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:12.945456 19181 server_base.cc:1034] running on GCE node
W20260810 01:47:12.945277 19190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:12.945749 19193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:12.946244 19181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:12.947104 19181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:12.948318 19181 hybrid_clock.cc:648] HybridClock initialized: now 1786326432948284 us; error 45 us; skew 500 ppm
Aug 10 01:47:12 dist-test-slave-kvfs krb5kdc[18817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326432, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:12.951622 19181 init.cc:377] Logged in from keytab as oryx/127.9.77.130@KRBTEST.COM (short username oryx)
I20260810 01:47:12.952764 19181 webserver.cc:460] Webserver started at http://127.9.77.130:35879/ using document root <none> and password file <none>
I20260810 01:47:12.953459 19181 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:12.953526 19181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:12.953845 19181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:12.956389 19181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/data/instance:
uuid: "12e99ab08944444b8ea106fa5aa38769"
format_stamp: "Formatted at 2026-08-10 01:47:12 on dist-test-slave-kvfs"
server_key: "568941a6858db70fc1e05205d1249742"
server_key_iv: "d0ed527293182d7ecc4550cda1f34bea"
server_key_version: "encryptionkey@0"
I20260810 01:47:12.957059 19181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance:
uuid: "12e99ab08944444b8ea106fa5aa38769"
format_stamp: "Formatted at 2026-08-10 01:47:12 on dist-test-slave-kvfs"
server_key: "568941a6858db70fc1e05205d1249742"
server_key_iv: "d0ed527293182d7ecc4550cda1f34bea"
server_key_version: "encryptionkey@0"
I20260810 01:47:12.961474 19181 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 01:47:12.964344 19203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:12.965801 19181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:47:12.965960 19181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "12e99ab08944444b8ea106fa5aa38769"
format_stamp: "Formatted at 2026-08-10 01:47:12 on dist-test-slave-kvfs"
server_key: "568941a6858db70fc1e05205d1249742"
server_key_iv: "d0ed527293182d7ecc4550cda1f34bea"
server_key_version: "encryptionkey@0"
I20260810 01:47:12.966082 19181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:12.991824 19181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:12.995826 19181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:12.996202 19181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:12.996994 19181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:12.998047 19181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:12.998123 19181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:12.998208 19181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:12.998253 19181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:13.009095 19181 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:46775
I20260810 01:47:13.009116 19369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:46775 every 8 connection(s)
I20260810 01:47:13.010238 19181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:47:13.014019 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 19181
I20260810 01:47:13.014175 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance
I20260810 01:47:13.014689 9526 external_mini_cluster.cc:1428] Setting key 7ca36b8cafa79d25ebca782ffb0ebd68
Aug 10 01:47:13 dist-test-slave-kvfs krb5kdc[18817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326432, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM
I20260810 01:47:13.029495 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:13.012526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:50047 (local address 127.9.77.190:33329)
0810 01:47:13.012753 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:13.012758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:13.013906 (+ 1148us) server_negotiation.cc:407] Connection header received
0810 01:47:13.015979 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:13.015983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:13.016100 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:13.016181 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:13.018213 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.018844 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.020887 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.021175 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.021346 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:13.025161 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:13.025165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:13.025167 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:13.025199 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:13.026920 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:13.027579 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:13.027582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:13.027584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:13.027677 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:13.028091 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:13.028095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:13.028096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:13.028336 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:13.028453 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:13.029130 (+ 677us) server_negotiation.cc:299] Negotiation successful
0810 01:47:13.029290 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
WARNING: no policy specified for oryx/127.9.77.131@KRBTEST.COM; defaulting to no policy
I20260810 01:47:13.030740 19374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:13.012993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.9.77.130:50047)
0810 01:47:13.013631 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:13.013745 (+ 114us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:13.015676 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:13.016453 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:13.016464 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:13.017062 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:13.018002 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:13.018035 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.019019 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.019027 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:13.020690 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:13.020730 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.021402 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.021408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:13.021646 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:13.022476 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:13.022511 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:13.024931 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:13.027103 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:13.027110 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:13.027131 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:13.027459 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:13.027791 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:13.027795 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:13.027796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:13.027965 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:13.028447 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:13.028454 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:13.028848 (+ 394us) client_negotiation.cc:769] Sending connection context
0810 01:47:13.029070 (+ 222us) client_negotiation.cc:240] Negotiation successful
0810 01:47:13.029500 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":144,"threads_started":1}
I20260810 01:47:13.032541 19370 heartbeater.cc:347] Connected to a master server at 127.9.77.190:33329
I20260810 01:47:13.032917 19370 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.77.131@KRBTEST.COM" created.
I20260810 01:47:13.033679 19370 heartbeater.cc:511] Master 127.9.77.190:33329 requested a full tablet report, sending...
I20260810 01:47:13.035195 18891 ts_manager.cc:194] Registered new tserver with Master: 12e99ab08944444b8ea106fa5aa38769 (127.9.77.130:46775)
I20260810 01:47:13.035971 18891 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.77.130@KRBTEST.COM'} at 127.9.77.130:50047
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.77.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.131.keytab.
Entry for principal oryx/127.9.77.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.131.keytab.
I20260810 01:47:13.053835 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:0
--local_ip_for_outbound_sockets=127.9.77.131
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:33329
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.131.keytab
--principal=oryx/127.9.77.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:13.190822 19377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:13.191205 19377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:13.191253 19377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:13.195538 19377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:13.195612 19377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:13.195710 19377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:47:13.201555 19377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.131.keytab
--principal=oryx/127.9.77.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:33329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:13.202955 19377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:13.204375 19377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:13.216332 19387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:13.216337 19391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:13.216338 19386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:13.216928 19377 server_base.cc:1034] running on GCE node
I20260810 01:47:13.217470 19377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:13.218297 19377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:13.219488 19377 hybrid_clock.cc:648] HybridClock initialized: now 1786326433219438 us; error 59 us; skew 500 ppm
Aug 10 01:47:13 dist-test-slave-kvfs krb5kdc[18817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326433, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:13.223002 19377 init.cc:377] Logged in from keytab as oryx/127.9.77.131@KRBTEST.COM (short username oryx)
I20260810 01:47:13.224167 19377 webserver.cc:460] Webserver started at http://127.9.77.131:34353/ using document root <none> and password file <none>
I20260810 01:47:13.224882 19377 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:13.224982 19377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:13.225258 19377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:13.227771 19377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/data/instance:
uuid: "49e30d0b090648f29deeb808001b14a7"
format_stamp: "Formatted at 2026-08-10 01:47:13 on dist-test-slave-kvfs"
server_key: "9c9075da294d5e67e00b5638439df55a"
server_key_iv: "e9375b1d7084c5cb02206d622cf0344d"
server_key_version: "encryptionkey@0"
I20260810 01:47:13.228498 19377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance:
uuid: "49e30d0b090648f29deeb808001b14a7"
format_stamp: "Formatted at 2026-08-10 01:47:13 on dist-test-slave-kvfs"
server_key: "9c9075da294d5e67e00b5638439df55a"
server_key_iv: "e9375b1d7084c5cb02206d622cf0344d"
server_key_version: "encryptionkey@0"
I20260810 01:47:13.233001 19377 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 01:47:13.235920 19400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:13.237402 19377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 01:47:13.237555 19377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "49e30d0b090648f29deeb808001b14a7"
format_stamp: "Formatted at 2026-08-10 01:47:13 on dist-test-slave-kvfs"
server_key: "9c9075da294d5e67e00b5638439df55a"
server_key_iv: "e9375b1d7084c5cb02206d622cf0344d"
server_key_version: "encryptionkey@0"
I20260810 01:47:13.237684 19377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:13.266289 19377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:13.270520 19377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:13.270851 19377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:13.271754 19377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:13.272951 19377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:13.273028 19377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:13.273100 19377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:13.273130 19377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:13.283702 19377 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:39543
I20260810 01:47:13.283723 19571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:39543 every 8 connection(s)
I20260810 01:47:13.284901 19377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:47:13.293489 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 19377
I20260810 01:47:13.293655 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance
I20260810 01:47:13.294014 9526 external_mini_cluster.cc:1428] Setting key b6ba5ff00367744dca217c1269b7df70
Aug 10 01:47:13 dist-test-slave-kvfs krb5kdc[18817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326433, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM
I20260810 01:47:13.301213 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:13.287315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:39827 (local address 127.9.77.190:33329)
0810 01:47:13.287611 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:13.287615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:13.288546 (+ 931us) server_negotiation.cc:407] Connection header received
0810 01:47:13.290227 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:13.290232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:13.290316 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:13.290431 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:13.291894 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.292573 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.294275 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.294541 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.294645 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:13.297578 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:13.297581 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:13.297583 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:13.297613 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:13.298983 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:13.299523 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:13.299526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:13.299528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:13.299616 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:13.299970 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:13.299972 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:13.299973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:13.300249 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:13.300334 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:13.300951 (+ 617us) server_negotiation.cc:299] Negotiation successful
0810 01:47:13.301070 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260810 01:47:13.302318 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:13.287788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.9.77.131:39827)
0810 01:47:13.288368 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:13.288421 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:13.289970 (+ 1549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:13.290651 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:13.290659 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:13.291108 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:13.291715 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:13.291731 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.292720 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.292724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:13.294120 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:13.294133 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.294745 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.294749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:13.294931 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:13.295637 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:13.295655 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:13.297415 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:13.299105 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:13.299111 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:13.299125 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:13.299428 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:13.299701 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:13.299704 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:13.299706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:13.299880 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:13.300378 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:13.300384 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:13.300682 (+ 298us) client_negotiation.cc:769] Sending connection context
0810 01:47:13.300910 (+ 228us) client_negotiation.cc:240] Negotiation successful
0810 01:47:13.301236 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":192,"threads_started":1}
I20260810 01:47:13.303666 19572 heartbeater.cc:347] Connected to a master server at 127.9.77.190:33329
I20260810 01:47:13.303980 19572 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:13.304628 19572 heartbeater.cc:511] Master 127.9.77.190:33329 requested a full tablet report, sending...
I20260810 01:47:13.305909 18891 ts_manager.cc:194] Registered new tserver with Master: 49e30d0b090648f29deeb808001b14a7 (127.9.77.131:39543)
I20260810 01:47:13.306603 18891 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.77.131@KRBTEST.COM'} at 127.9.77.131:39827
I20260810 01:47:13.308908 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.77.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.77.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.77.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.189.keytab.
Entry for principal oryx/127.9.77.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.189.keytab.
I20260810 01:47:13.344111 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.189:38111
--webserver_interface=127.9.77.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.189.keytab
--principal=oryx/127.9.77.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:33329,127.9.77.189:38111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:13.479079 19581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:13.479429 19581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:13.479475 19581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:13.484316 19581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:47:13.484453 19581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:13.484496 19581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:13.484521 19581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:47:13.484549 19581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:47:13.490775 19581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:33329,127.9.77.189:38111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.189.keytab
--principal=oryx/127.9.77.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.189:38111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.77.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
W20260810 01:47:13.491119 19581 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.9.77.190:33329,127.9.77.189:38111'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 01:47:13.492470 19581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:13.493791 19581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:13.504185 19594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:13.504185 19592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:13.504225 19591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:13.504503 19581 server_base.cc:1034] running on GCE node
I20260810 01:47:13.505287 19581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:13.506788 19581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:13.508064 19581 hybrid_clock.cc:648] HybridClock initialized: now 1786326433507979 us; error 49 us; skew 500 ppm
Aug 10 01:47:13 dist-test-slave-kvfs krb5kdc[18817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326433, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:13.511746 19581 init.cc:377] Logged in from keytab as oryx/127.9.77.189@KRBTEST.COM (short username oryx)
I20260810 01:47:13.513027 19581 webserver.cc:460] Webserver started at http://127.9.77.189:36465/ using document root <none> and password file <none>
I20260810 01:47:13.513746 19581 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:13.513849 19581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:13.514133 19581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:13.516711 19581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-1/data/instance:
uuid: "b3e994dc2a3d412ea8f13f43c39bc783"
format_stamp: "Formatted at 2026-08-10 01:47:13 on dist-test-slave-kvfs"
server_key: "2eda0f1b13c22a6b7748ed675bc56fac"
server_key_iv: "591074fb22770106153029d8fcb0c3d2"
server_key_version: "encryptionkey@0"
I20260810 01:47:13.517380 19581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-1/wal/instance:
uuid: "b3e994dc2a3d412ea8f13f43c39bc783"
format_stamp: "Formatted at 2026-08-10 01:47:13 on dist-test-slave-kvfs"
server_key: "2eda0f1b13c22a6b7748ed675bc56fac"
server_key_iv: "591074fb22770106153029d8fcb0c3d2"
server_key_version: "encryptionkey@0"
I20260810 01:47:13.521945 19581 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 01:47:13.525194 19605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:13.526844 19581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:47:13.527025 19581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-1/wal
uuid: "b3e994dc2a3d412ea8f13f43c39bc783"
format_stamp: "Formatted at 2026-08-10 01:47:13 on dist-test-slave-kvfs"
server_key: "2eda0f1b13c22a6b7748ed675bc56fac"
server_key_iv: "591074fb22770106153029d8fcb0c3d2"
server_key_version: "encryptionkey@0"
I20260810 01:47:13.527165 19581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:13.563124 19581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:13.567428 19581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:13.567723 19581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:13.577018 19581 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.189:38111
I20260810 01:47:13.577371 19688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.189:38111 every 8 connection(s)
I20260810 01:47:13.578308 19581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-1/data/info.pb
I20260810 01:47:13.581923 19689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:13.583956 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 19581
I20260810 01:47:13.587253 19689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 01:47:13 dist-test-slave-kvfs krb5kdc[18817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.77.189@KRBTEST.COM
Aug 10 01:47:13 dist-test-slave-kvfs krb5kdc[18817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326433, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.189@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM
I20260810 01:47:13.599953 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:13.589449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36558 (local address 127.9.77.190:33329)
0810 01:47:13.589631 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:13.589636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:13.590047 (+ 411us) server_negotiation.cc:407] Connection header received
0810 01:47:13.590186 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:13.590190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:13.590256 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:13.590352 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:13.591130 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.591700 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.592776 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.593024 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.593160 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:13.596061 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:13.596067 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:13.596072 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:13.596111 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:13.597669 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:13.598204 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:13.598208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:13.598210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:13.598307 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:13.598666 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:13.598669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:13.598671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:13.599004 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:13.599128 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:13.599612 (+ 484us) server_negotiation.cc:299] Negotiation successful
0810 01:47:13.599751 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 01:47:13.600943 19693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:13.586030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45760 (local address 127.9.77.189:38111)
0810 01:47:13.586553 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:13.586572 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:13.586616 (+ 44us) server_negotiation.cc:407] Connection header received
0810 01:47:13.587353 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:13.587375 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:13.587856 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:13.588309 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:13.589262 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.590680 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.591609 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.592227 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.592406 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:13.595512 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:13.595543 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:13.595560 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:13.595586 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:13.597442 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:13.597978 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:13.597983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:13.597985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:13.598104 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:13.598475 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:13.598478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:13.598479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:13.599026 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:13.599244 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:13.599558 (+ 314us) server_negotiation.cc:299] Negotiation successful
0810 01:47:13.599877 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":155,"threads_started":1}
I20260810 01:47:13.600940 19694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:13.589330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.0.0.1:36558)
0810 01:47:13.589885 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:13.589939 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:13.590076 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:13.590440 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:13.590445 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:13.590840 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:13.590971 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:13.590976 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.591891 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.591894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:13.592631 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:13.592641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.593206 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.593211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:13.593332 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:13.593925 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:13.593941 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:13.595873 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:13.597778 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:13.597789 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:13.597802 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:13.598086 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:13.598419 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:13.598425 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:13.598428 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:13.598563 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:13.599171 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:13.599179 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:13.599352 (+ 173us) client_negotiation.cc:769] Sending connection context
0810 01:47:13.599549 (+ 197us) client_negotiation.cc:240] Negotiation successful
0810 01:47:13.600059 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":1,"thread_start_us":191,"threads_started":1}
I20260810 01:47:13.602576 19689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 01:47:13 dist-test-slave-kvfs krb5kdc[18817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326433, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.189@KRBTEST.COM for oryx/127.9.77.189@KRBTEST.COM
I20260810 01:47:13.611550 19694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:13.603170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:38111 (local address 127.0.0.1:45774)
0810 01:47:13.603433 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:13.603448 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:13.603549 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:13.603805 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:13.603807 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:13.603950 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:13.604110 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:13.604117 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.605230 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.605232 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:13.605758 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:13.605766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.606267 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.606271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:13.606353 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:13.606839 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:13.606851 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:13.608335 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:13.610058 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:13.610061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:13.610063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:13.610273 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:13.610603 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:13.610605 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:13.610607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:13.610692 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:13.611129 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:13.611131 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:13.611210 (+ 79us) client_negotiation.cc:769] Sending connection context
0810 01:47:13.611295 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 01:47:13.611409 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195}
I20260810 01:47:13.611635 19693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:13.603302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45774 (local address 127.9.77.189:38111)
0810 01:47:13.603429 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:13.603433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:13.603520 (+ 87us) server_negotiation.cc:407] Connection header received
0810 01:47:13.603643 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:13.603645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:13.603681 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:13.603746 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:13.604207 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.605111 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.605867 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.606161 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.606242 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:13.608478 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:13.608482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:13.608484 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:13.608504 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:13.609928 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:13.610381 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:13.610384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:13.610385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:13.610481 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:13.610811 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:13.610814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:13.610815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:13.611032 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:13.611131 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:13.611369 (+ 238us) server_negotiation.cc:299] Negotiation successful
0810 01:47:13.611477 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 01:47:13.615477 19689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783: Bootstrap starting.
I20260810 01:47:13.618700 19689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:13.619704 19689 log.cc:824] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:13.622380 19689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783: No bootstrap required, opened a new log
I20260810 01:47:13.626039 19689 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } }
I20260810 01:47:13.626330 19689 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:13.626416 19689 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e994dc2a3d412ea8f13f43c39bc783, State: Initialized, Role: FOLLOWER
I20260810 01:47:13.626993 19689 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } }
I20260810 01:47:13.627609 19701 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } } }
I20260810 01:47:13.627750 19701 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 01:47:13.628638 19689 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 01:47:13.634336 19719 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 01:47:13.634447 19719 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 01:47:13.637341 19581 master_runner.cc:428] Detected that this master b3e994dc2a3d412ea8f13f43c39bc783 is joining an existing cluster
I20260810 01:47:13.637418 19581 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.77.189:38111
I20260810 01:47:13.649883 19726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:13.640403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.0.0.1:36566)
0810 01:47:13.640807 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:13.640823 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:13.640946 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:13.641343 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:13.641346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:13.641683 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:13.641861 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:13.641871 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.642944 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.642950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:13.643767 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:13.643777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.644381 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.644385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:13.644486 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:13.645170 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:13.645184 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:13.645599 (+ 415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:13.647862 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:13.647867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:13.647869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:13.648238 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:13.648590 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:13.648594 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:13.648597 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:13.648703 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:13.649228 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:13.649231 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:13.649338 (+ 107us) client_negotiation.cc:769] Sending connection context
0810 01:47:13.649445 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 01:47:13.649601 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":128,"threads_started":1}
I20260810 01:47:13.649922 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:13.640482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36566 (local address 127.9.77.190:33329)
0810 01:47:13.640661 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:13.640667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:13.640927 (+ 260us) server_negotiation.cc:407] Connection header received
0810 01:47:13.641063 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:13.641067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:13.641135 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:13.641261 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:13.642017 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.642827 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.643895 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.644240 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.644370 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:13.645774 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:13.645778 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:13.645781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:13.645819 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:13.647742 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:13.648358 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:13.648362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:13.648364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:13.648473 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:13.648846 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:13.648850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:13.648852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:13.649121 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:13.649265 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:13.649532 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 01:47:13.649693 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 01:47:13.651163 19693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:13.640616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45786 (local address 127.9.77.189:38111)
0810 01:47:13.640734 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:13.640739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:13.641235 (+ 496us) server_negotiation.cc:407] Connection header received
0810 01:47:13.641404 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:13.641408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:13.641445 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:13.641497 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:13.642094 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.642932 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.644778 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.645161 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.645236 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:13.646649 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:13.646653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:13.646657 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:13.646681 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:13.648660 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:13.649162 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:13.649169 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:13.649174 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:13.649268 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:13.649571 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:13.649574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:13.649576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:13.649794 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:13.649871 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:13.650842 (+ 971us) server_negotiation.cc:299] Negotiation successful
0810 01:47:13.650995 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":35}
I20260810 01:47:13.651165 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:13.640403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:38111 (local address 127.0.0.1:45786)
0810 01:47:13.641145 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:13.641160 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:13.641306 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:13.641601 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:13.641603 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:13.641833 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:13.641986 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:13.641997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.643120 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.643123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:13.644623 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:13.644638 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.645376 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.645379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:13.645468 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:13.646005 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:13.646019 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:13.646502 (+ 483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:13.648781 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:13.648785 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:13.648787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:13.649051 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:13.649355 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:13.649361 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:13.649363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:13.649464 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:13.650527 (+ 1063us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:13.650533 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:13.650637 (+ 104us) client_negotiation.cc:769] Sending connection context
0810 01:47:13.650764 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 01:47:13.650920 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":25,"thread_start_us":138,"threads_started":1}
Aug 10 01:47:13 dist-test-slave-kvfs krb5kdc[18817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326432, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.190@KRBTEST.COM for oryx/127.9.77.189@KRBTEST.COM
WARNING: no policy specified for oryx/127.9.77.188@KRBTEST.COM; defaulting to no policy
I20260810 01:47:13.668540 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:13.655919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:38111 (local address 127.0.0.1:45788)
0810 01:47:13.656864 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:13.656912 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:13.657093 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:13.657454 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:13.657461 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:13.658036 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:13.658253 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:13.658265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.659347 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.659351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:13.660378 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:13.660390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.660932 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.660935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:13.661027 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:13.661687 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:13.661701 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:13.663595 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:13.665938 (+ 2343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:13.665953 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:13.665964 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:13.666392 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:13.666739 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:13.666745 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:13.666750 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:13.666873 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:13.667387 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:13.667400 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:13.667636 (+ 236us) client_negotiation.cc:769] Sending connection context
0810 01:47:13.667858 (+ 222us) client_negotiation.cc:240] Negotiation successful
0810 01:47:13.668213 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":159,"threads_started":1}
I20260810 01:47:13.668576 19693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:13.655919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45788 (local address 127.9.77.189:38111)
0810 01:47:13.656103 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:13.656109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:13.657004 (+ 895us) server_negotiation.cc:407] Connection header received
0810 01:47:13.657246 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:13.657250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:13.657315 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:13.657413 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:13.658398 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.659213 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.660512 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.660815 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.660922 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:13.663807 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:13.663812 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:13.663815 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:13.663846 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:13.665786 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:13.666522 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:13.666526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:13.666528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:13.666626 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:13.666988 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:13.666991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:13.666993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:13.667245 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:13.667353 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:13.667933 (+ 580us) server_negotiation.cc:299] Negotiation successful
0810 01:47:13.668355 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 01:47:13.671031 18891 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.77.189:38111: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783"
member_type: NON_VOTER
last_known_addr {
host: "127.9.77.189"
port: 38111
}
attrs {
promote: true
}
}
dest_uuid: "9ae815b0781f4d19aba6d526311d18c9"
cas_config_opid_index: -1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.77.188@KRBTEST.COM" created.
I20260810 01:47:13.672267 18891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: NON_VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: true } }
I20260810 01:47:13.674804 19734 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b3e994dc2a3d412ea8f13f43c39bc783 (127.9.77.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: NON_VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: true } } }
I20260810 01:47:13.675657 19650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:13.677074 19734 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.77.189:38111
I20260810 01:47:13.677436 19650 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 1 FOLLOWER]: Refusing update from remote peer 9ae815b0781f4d19aba6d526311d18c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 01:47:13.677814 19581 master.cc:537] Master@127.9.77.189:38111 shutting down...
I20260810 01:47:13.677508 19734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9ae815b0781f4d19aba6d526311d18c9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: NON_VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: true } } }
I20260810 01:47:13.677973 19734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:13.680146 19734 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: NON_VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 01:47:13.682993 18868 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260810 01:47:13.684001 19581 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260810 01:47:13.684403 18868 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 -> Peer b3e994dc2a3d412ea8f13f43c39bc783 (127.9.77.189:38111): Couldn't send request to peer b3e994dc2a3d412ea8f13f43c39bc783. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 01:47:13.684655 19581 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:47:13.684830 19581 tablet_replica.cc:330] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.77.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.188.keytab.
Entry for principal oryx/127.9.77.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.188.keytab.
I20260810 01:47:13.701004 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.188:45735
--webserver_interface=127.9.77.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.188.keytab
--principal=oryx/127.9.77.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:33329,127.9.77.189:38111,127.9.77.188:45735
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 01:47:13.702574 19581 master.cc:552] Master@127.9.77.189:38111 shutdown complete.
I20260810 01:47:13.702656 19581 master_runner.cc:305] Clearing existing system tablet
I20260810 01:47:13.703015 19581 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:13.703091 19581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:13.704931 19581 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.004s sys 0.000s
I20260810 01:47:13.706230 19743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:13.706599 19581 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:13.706775 19581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 01:47:13.706820 19581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-1/wal
uuid: "b3e994dc2a3d412ea8f13f43c39bc783"
format_stamp: "Formatted at 2026-08-10 01:47:13 on dist-test-slave-kvfs"
server_key: "2eda0f1b13c22a6b7748ed675bc56fac"
server_key_iv: "591074fb22770106153029d8fcb0c3d2"
server_key_version: "encryptionkey@0"
I20260810 01:47:13.709578 19581 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:47:13.712114 19581 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 01:47:13.712242 19581 log.cc:1196] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 01:47:13.712584 19581 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783: Deleting consensus metadata
I20260810 01:47:13.712815 19581 master_runner.cc:315] Copying system tablet from 127.9.77.190:33329
I20260810 01:47:13.714321 19581 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783: tablet copy: Beginning tablet copy session from remote peer at address 127.9.77.190:33329
I20260810 01:47:13.725071 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:13.715226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36578 (local address 127.9.77.190:33329)
0810 01:47:13.715417 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:13.715423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:13.715696 (+ 273us) server_negotiation.cc:407] Connection header received
0810 01:47:13.715907 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:13.715917 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:13.715981 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:13.716120 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:13.717047 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.718114 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.719231 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.719507 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.719641 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:13.721103 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:13.721108 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:13.721111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:13.721151 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:13.723086 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:13.723593 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:13.723597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:13.723599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:13.723694 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:13.724063 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:13.724066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:13.724068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:13.724295 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:13.724434 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:13.724722 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 01:47:13.724857 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 01:47:13.725080 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:13.715085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.0.0.1:36578)
0810 01:47:13.715569 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:13.715584 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:13.715759 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:13.716399 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:13.716402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:13.716671 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:13.716851 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:13.716862 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.718299 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.718302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:13.719096 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:13.719107 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.719650 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.719654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:13.719773 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:13.720494 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:13.720509 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:13.720932 (+ 423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:13.723236 (+ 2304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:13.723239 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:13.723241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:13.723475 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:13.723804 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:13.723807 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:13.723809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:13.723936 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:13.724418 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:13.724421 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:13.724540 (+ 119us) client_negotiation.cc:769] Sending connection context
0810 01:47:13.724655 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 01:47:13.724855 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":142,"threads_started":1}
I20260810 01:47:13.726147 18922 tablet_copy_service.cc:140] P 9ae815b0781f4d19aba6d526311d18c9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b3e994dc2a3d412ea8f13f43c39bc783 ({username='oryx', principal='oryx/127.9.77.189@KRBTEST.COM'} at 127.0.0.1:36578)
I20260810 01:47:13.726397 18922 tablet_copy_service.cc:161] P 9ae815b0781f4d19aba6d526311d18c9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b3e994dc2a3d412ea8f13f43c39bc783 at {username='oryx', principal='oryx/127.9.77.189@KRBTEST.COM'} at 127.0.0.1:36578: session id = b3e994dc2a3d412ea8f13f43c39bc783-00000000000000000000000000000000
I20260810 01:47:13.728425 18922 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 01:47:13.730903 19581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 01:47:13.733405 19581 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783: tablet copy: Starting download of 0 data blocks...
I20260810 01:47:13.733659 19581 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783: tablet copy: Starting download of 1 WAL segments...
I20260810 01:47:13.736141 19581 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 01:47:13.741770 18922 tablet_copy_service.cc:342] P 9ae815b0781f4d19aba6d526311d18c9: Request end of tablet copy session b3e994dc2a3d412ea8f13f43c39bc783-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.77.189@KRBTEST.COM'} at 127.0.0.1:36578
I20260810 01:47:13.742250 18922 tablet_copy_service.cc:433] P 9ae815b0781f4d19aba6d526311d18c9: ending tablet copy session b3e994dc2a3d412ea8f13f43c39bc783-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b3e994dc2a3d412ea8f13f43c39bc783
I20260810 01:47:13.752105 19581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:13.754238 19767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:13.754185 19763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:13.754963 19581 server_base.cc:1034] running on GCE node
W20260810 01:47:13.756220 19764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:13.756521 19581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:13.756933 19581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:13.758096 19581 hybrid_clock.cc:648] HybridClock initialized: now 1786326433758074 us; error 43 us; skew 500 ppm
Aug 10 01:47:13 dist-test-slave-kvfs krb5kdc[18817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326433, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:13.760139 19581 init.cc:377] Logged in from keytab as oryx/127.9.77.189@KRBTEST.COM (short username oryx)
I20260810 01:47:13.760746 19581 webserver.cc:460] Webserver started at http://127.9.77.189:40683/ using document root <none> and password file <none>
I20260810 01:47:13.760985 19581 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:13.761044 19581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:13.762573 19581 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 01:47:13.763703 19780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:13.763970 19581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 01:47:13.764149 19581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-1/wal
uuid: "b3e994dc2a3d412ea8f13f43c39bc783"
format_stamp: "Formatted at 2026-08-10 01:47:13 on dist-test-slave-kvfs"
server_key: "2eda0f1b13c22a6b7748ed675bc56fac"
server_key_iv: "591074fb22770106153029d8fcb0c3d2"
server_key_version: "encryptionkey@0"
I20260810 01:47:13.764283 19581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:13.767495 19176 heartbeater.cc:503] Master 127.9.77.190:33329 was elected leader, sending a full tablet report...
I20260810 01:47:13.775501 19581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:13.777876 19581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:13.783686 19581 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.189:38111
I20260810 01:47:13.783699 19864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.189:38111 every 8 connection(s)
I20260810 01:47:13.784437 19581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-1/data/info.pb
I20260810 01:47:13.786355 19865 sys_catalog.cc:263] Verifying existing consensus state
I20260810 01:47:13.787320 19865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783: Bootstrap starting.
I20260810 01:47:13.797734 19865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 01:47:13.798198 19865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783: Bootstrap complete.
I20260810 01:47:13.799165 19865 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: NON_VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: true } }
I20260810 01:47:13.799351 19865 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3e994dc2a3d412ea8f13f43c39bc783, State: Initialized, Role: LEARNER
I20260810 01:47:13.799489 19865 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: NON_VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: true } }
I20260810 01:47:13.799925 19872 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: NON_VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: true } } }
I20260810 01:47:13.800135 19865 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:47:13.800143 19872 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [sys.catalog]: This master's current role is: LEARNER
I20260810 01:47:13.804221 19892 catalog_manager.cc:1259] Loaded cluster ID: 30255652128446de977d487ec2e58d4b
I20260810 01:47:13.804323 19892 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783: loading cluster ID for follower catalog manager: success
I20260810 01:47:13.806680 19892 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783: acquiring CA information for follower catalog manager: success
I20260810 01:47:13.808379 19892 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 01:47:13.879640 19739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:13.879995 19739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:13.880065 19739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:13.884563 19739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:47:13.884657 19739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:13.884703 19739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:13.884748 19739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:47:13.884794 19739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:47:13.895092 19739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:33329,127.9.77.189:38111,127.9.77.188:45735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.188.keytab
--principal=oryx/127.9.77.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.188:45735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.77.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:13.897176 19739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:13.898803 19739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:13.910967 19900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:13.911065 19739 server_base.cc:1034] running on GCE node
W20260810 01:47:13.910964 19901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:13.910967 19904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:13.912106 19739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:13.913635 19739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:13.914834 19739 hybrid_clock.cc:648] HybridClock initialized: now 1786326433914816 us; error 32 us; skew 500 ppm
Aug 10 01:47:13 dist-test-slave-kvfs krb5kdc[18817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326433, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:13.918313 19739 init.cc:377] Logged in from keytab as oryx/127.9.77.188@KRBTEST.COM (short username oryx)
I20260810 01:47:13.919605 19739 webserver.cc:460] Webserver started at http://127.9.77.188:38905/ using document root <none> and password file <none>
I20260810 01:47:13.920444 19739 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:13.920552 19739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:13.920862 19739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:13.923692 19739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-2/data/instance:
uuid: "646e25980e154c5c8b8c24f9040a9bb5"
format_stamp: "Formatted at 2026-08-10 01:47:13 on dist-test-slave-kvfs"
server_key: "2678b268cdba301751837f2936516c77"
server_key_iv: "fb8d6d88e4bb332baebf54af255e4336"
server_key_version: "encryptionkey@0"
I20260810 01:47:13.924556 19739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-2/wal/instance:
uuid: "646e25980e154c5c8b8c24f9040a9bb5"
format_stamp: "Formatted at 2026-08-10 01:47:13 on dist-test-slave-kvfs"
server_key: "2678b268cdba301751837f2936516c77"
server_key_iv: "fb8d6d88e4bb332baebf54af255e4336"
server_key_version: "encryptionkey@0"
I20260810 01:47:13.929606 19739 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 01:47:13.932840 19916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:13.934458 19739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:47:13.934640 19739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-2/wal
uuid: "646e25980e154c5c8b8c24f9040a9bb5"
format_stamp: "Formatted at 2026-08-10 01:47:13 on dist-test-slave-kvfs"
server_key: "2678b268cdba301751837f2936516c77"
server_key_iv: "fb8d6d88e4bb332baebf54af255e4336"
server_key_version: "encryptionkey@0"
I20260810 01:47:13.934782 19739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:13.965866 19739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:13.970081 19739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:13.970326 19739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:13.979564 19739 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.188:45735
I20260810 01:47:13.979939 19996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.188:45735 every 8 connection(s)
I20260810 01:47:13.980875 19739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-2/data/info.pb
I20260810 01:47:13.983907 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 19739
I20260810 01:47:13.984537 19997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:13.990290 19997 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 01:47:13 dist-test-slave-kvfs krb5kdc[18817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.77.188@KRBTEST.COM
Aug 10 01:47:13 dist-test-slave-kvfs krb5kdc[18817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326433, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.188@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM
I20260810 01:47:14.003247 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:13.985898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51022 (local address 127.9.77.188:45735)
0810 01:47:13.986468 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:13.986488 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:13.986535 (+ 47us) server_negotiation.cc:407] Connection header received
0810 01:47:13.987389 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:13.987411 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:13.987856 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:13.988359 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:13.989457 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.990751 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.991822 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.992583 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.993362 (+ 779us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:13.996718 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:13.996724 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:13.996736 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:13.996769 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:13.998996 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:13.999550 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:13.999556 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:13.999562 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:13.999683 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.000016 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.000047 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.000049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.000702 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:14.000975 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:14.001357 (+ 382us) server_negotiation.cc:299] Negotiation successful
0810 01:47:14.001766 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":172,"threads_started":1}
I20260810 01:47:14.003521 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:13.992396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.0.0.1:36584)
0810 01:47:13.992825 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:13.992899 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:13.993052 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:13.993430 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:13.993437 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:13.993896 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:13.994040 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:13.994046 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.995283 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.995288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:13.996390 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:13.996403 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.996969 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.996973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:13.997131 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:13.997651 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:13.997673 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:13.999472 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:14.001573 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.001584 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.001591 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.001883 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.002215 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.002218 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.002219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.002334 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.002832 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:14.002838 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:14.002966 (+ 128us) client_negotiation.cc:769] Sending connection context
0810 01:47:14.003089 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 01:47:14.003248 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":155,"threads_started":1}
I20260810 01:47:14.003539 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:13.992693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36584 (local address 127.9.77.190:33329)
0810 01:47:13.992887 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:13.992892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:13.993006 (+ 114us) server_negotiation.cc:407] Connection header received
0810 01:47:13.993167 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:13.993174 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:13.993252 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:13.993351 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:13.994206 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.995081 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.996539 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:13.996835 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:13.996922 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:13.999636 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:13.999639 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:13.999642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:13.999670 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:14.001419 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.001995 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.001999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.002001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.002104 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.002450 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.002453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.002455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.002712 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:14.002824 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:14.003184 (+ 360us) server_negotiation.cc:299] Negotiation successful
0810 01:47:14.003365 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 01:47:14.004918 19997 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 01:47:14 dist-test-slave-kvfs krb5kdc[18817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326433, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.188@KRBTEST.COM for oryx/127.9.77.189@KRBTEST.COM
I20260810 01:47:14.014741 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.005524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:38111 (local address 127.0.0.1:45804)
0810 01:47:14.005695 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:14.005707 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:14.005791 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:14.006389 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:14.006392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:14.006549 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:14.006657 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.006662 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.007670 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.007673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.008375 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.008385 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.008893 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.008898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.008993 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.009562 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:14.009575 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:14.011022 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:14.013301 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.013304 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.013305 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.013518 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.013810 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.013812 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.013813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.013889 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.014329 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:14.014331 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:14.014413 (+ 82us) client_negotiation.cc:769] Sending connection context
0810 01:47:14.014494 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 01:47:14.014609 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260810 01:47:14.014866 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.005689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45804 (local address 127.9.77.189:38111)
0810 01:47:14.006145 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:14.006149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:14.006162 (+ 13us) server_negotiation.cc:407] Connection header received
0810 01:47:14.006208 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:14.006211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:14.006259 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:14.006341 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:14.006775 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.007567 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.008488 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.008764 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.008873 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.011182 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:14.011186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:14.011189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:14.011220 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:14.013184 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.013613 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.013617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.013619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.013718 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.013980 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.013983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.013985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.014226 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:14.014333 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:14.014561 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 01:47:14.014680 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":220,"threads_started":1}
I20260810 01:47:14.015762 19997 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.188" port: 45735 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 01:47:14 dist-test-slave-kvfs krb5kdc[18817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326433, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.188@KRBTEST.COM for oryx/127.9.77.188@KRBTEST.COM
I20260810 01:47:14.024726 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.016456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:45735 (local address 127.0.0.1:51028)
0810 01:47:14.016617 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:14.016635 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:14.016726 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:14.016991 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:14.016994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:14.017169 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:14.017233 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.017238 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.018414 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.018417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.018931 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.018939 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.019381 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.019385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.019475 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.019981 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:14.019992 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:14.021458 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:14.023123 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.023129 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.023131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.023337 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.023615 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.023618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.023620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.023698 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.024152 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:14.024155 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:14.024250 (+ 95us) client_negotiation.cc:769] Sending connection context
0810 01:47:14.024365 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 01:47:14.024521 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":1}
I20260810 01:47:14.024726 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.016516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51028 (local address 127.9.77.188:45735)
0810 01:47:14.016618 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:14.016624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:14.016683 (+ 59us) server_negotiation.cc:407] Connection header received
0810 01:47:14.016829 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:14.016832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:14.016873 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:14.016974 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:14.017341 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.018294 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.019044 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.019274 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.019383 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.021611 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:14.021614 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:14.021616 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:14.021636 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:14.023018 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.023436 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.023441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.023443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.023519 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.023799 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.023802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.023804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.023986 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:14.024162 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:14.024422 (+ 260us) server_negotiation.cc:299] Negotiation successful
0810 01:47:14.024590 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":47}
I20260810 01:47:14.028729 19997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5: Bootstrap starting.
I20260810 01:47:14.031817 19997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:14.032774 19997 log.cc:824] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:14.035250 19997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5: No bootstrap required, opened a new log
I20260810 01:47:14.038736 19370 heartbeater.cc:503] Master 127.9.77.190:33329 was elected leader, sending a full tablet report...
I20260810 01:47:14.039117 19997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 45735 } }
I20260810 01:47:14.039474 19997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:14.039575 19997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 646e25980e154c5c8b8c24f9040a9bb5, State: Initialized, Role: FOLLOWER
I20260810 01:47:14.040198 19997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 45735 } }
I20260810 01:47:14.040942 20004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 45735 } } }
I20260810 01:47:14.041138 20004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 01:47:14.041920 19997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 01:47:14.048300 20016 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 01:47:14.048448 20016 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 01:47:14.052184 19739 master_runner.cc:428] Detected that this master 646e25980e154c5c8b8c24f9040a9bb5 is joining an existing cluster
I20260810 01:47:14.052253 19739 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.77.188:45735
I20260810 01:47:14.057430 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 18984
I20260810 01:47:14.067874 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.055366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36590 (local address 127.9.77.190:33329)
0810 01:47:14.055514 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:14.055519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:14.055924 (+ 405us) server_negotiation.cc:407] Connection header received
0810 01:47:14.056409 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:14.056414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:14.056475 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:14.056555 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:14.057201 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.057920 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.062510 (+ 4590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.062949 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.063040 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.064602 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:14.064606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:14.064609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:14.064636 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:14.066175 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.066626 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.066629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.066631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.066707 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.066953 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.066956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.066957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.067157 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:14.067223 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:14.067569 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 01:47:14.067697 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 01:47:14.068723 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.054796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.0.0.1:36590)
0810 01:47:14.055798 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:14.055815 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:14.055951 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:14.056691 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:14.056694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:14.056914 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:14.057052 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.057060 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.061157 (+ 4097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.061161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.062340 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.062352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.063225 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.063228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.063316 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.063947 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:14.063962 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:14.064456 (+ 494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:14.066310 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.066313 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.066315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.066532 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.066800 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.066802 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.066804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.066874 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.067295 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:14.067297 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:14.067384 (+ 87us) client_negotiation.cc:769] Sending connection context
0810 01:47:14.068411 (+ 1027us) client_negotiation.cc:240] Negotiation successful
0810 01:47:14.068555 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"thread_start_us":157,"threads_started":1}
I20260810 01:47:14.104229 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.055596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51044 (local address 127.9.77.188:45735)
0810 01:47:14.055704 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:14.055708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:14.069488 (+ 13780us) server_negotiation.cc:407] Connection header received
0810 01:47:14.069622 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:14.069625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:14.069664 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:14.069711 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:14.070198 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.070937 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.074119 (+ 3182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.074455 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.074546 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.098201 (+ 23655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:14.098212 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:14.098218 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:14.098259 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:14.100859 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.102026 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.102032 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.102037 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.102156 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.102547 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.102554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.102558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.103051 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:14.103165 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:14.103616 (+ 451us) server_negotiation.cc:299] Negotiation successful
0810 01:47:14.103820 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 01:47:14.105000 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.055471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:45735 (local address 127.0.0.1:51044)
0810 01:47:14.069431 (+ 13960us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:14.069445 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:14.069537 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:14.069820 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:14.069823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:14.070008 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:14.070088 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.070094 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.071070 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.071072 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.073999 (+ 2927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.074010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.074687 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.074691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.074793 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.097102 (+ 22309us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:14.097135 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:14.097916 (+ 781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:14.101505 (+ 3589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.101512 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.101518 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.101884 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.102289 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.102295 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.102300 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.102425 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.103231 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:14.103237 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:14.103379 (+ 142us) client_negotiation.cc:769] Sending connection context
0810 01:47:14.104470 (+ 1091us) client_negotiation.cc:240] Negotiation successful
0810 01:47:14.104694 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13902,"thread_start_us":116,"threads_started":1}
Aug 10 01:47:14 dist-test-slave-kvfs krb5kdc[18817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326432, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.190@KRBTEST.COM for oryx/127.9.77.188@KRBTEST.COM
I20260810 01:47:14.113801 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.074622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:45735 (local address 127.0.0.1:51060)
0810 01:47:14.074919 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:14.074934 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:14.075015 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:14.075827 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:14.075832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:14.076094 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:14.076243 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.076253 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.077458 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.077464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.078362 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.078376 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.079008 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.079014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.079151 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.079692 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:14.079710 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:14.108794 (+ 29084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:14.111227 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.111232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.111234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.111565 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.112134 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.112138 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.112141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.112311 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.112937 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:14.112942 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:14.113118 (+ 176us) client_negotiation.cc:769] Sending connection context
0810 01:47:14.113278 (+ 160us) client_negotiation.cc:240] Negotiation successful
0810 01:47:14.113542 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204}
I20260810 01:47:14.114939 20032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.075131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51060 (local address 127.9.77.188:45735)
0810 01:47:14.075514 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:14.075524 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:14.075538 (+ 14us) server_negotiation.cc:407] Connection header received
0810 01:47:14.075617 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:14.075622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:14.075684 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:14.076363 (+ 679us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:14.076483 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.077336 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.078513 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.078892 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.080424 (+ 1532us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.108984 (+ 28560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:14.108991 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:14.108997 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:14.109031 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:14.111079 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.111700 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.111703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.111705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.111794 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.112475 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.112481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.112489 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.112788 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:14.114409 (+ 1621us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:14.114496 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 01:47:14.114668 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":211,"threads_started":1}
I20260810 01:47:14.116308 18891 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.77.188:45735: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5"
member_type: NON_VOTER
last_known_addr {
host: "127.9.77.188"
port: 45735
}
attrs {
promote: true
}
}
dest_uuid: "9ae815b0781f4d19aba6d526311d18c9"
cas_config_opid_index: 5
I20260810 01:47:14.116891 18891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: NON_VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: true } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: NON_VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: true } }
I20260810 01:47:14.118396 19734 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 646e25980e154c5c8b8c24f9040a9bb5 (127.9.77.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: NON_VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: true } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: NON_VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: true } } }
I20260810 01:47:14.119057 19964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:14.120213 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:45485
--local_ip_for_outbound_sockets=127.9.77.129
--tserver_master_addrs=127.9.77.190:33329
--webserver_port=37359
--webserver_interface=127.9.77.129
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.129.keytab
--principal=oryx/127.9.77.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.77.190:33329,127.9.77.189:38111
--tserver_master_addrs=127.9.77.190:33329,127.9.77.189:38111,127.9.77.188:45735 with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 01:47:14.120442 19734 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.77.188:45735
I20260810 01:47:14.120672 19734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9ae815b0781f4d19aba6d526311d18c9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: NON_VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: true } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: NON_VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: true } } }
I20260810 01:47:14.120807 19734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:14.121065 19964 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [term 1 FOLLOWER]: Refusing update from remote peer 9ae815b0781f4d19aba6d526311d18c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 01:47:14.121354 19739 master.cc:537] Master@127.9.77.188:45735 shutting down...
I20260810 01:47:14.121521 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.113495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:38111 (local address 127.0.0.1:45816)
0810 01:47:14.114024 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:14.114044 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:14.114139 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:14.116854 (+ 2715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:14.116858 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:14.116887 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:14.117045 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.117054 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.118358 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.118361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.119757 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.119774 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.120973 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.120981 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.121110 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.121186 (+ 76us) client_negotiation.cc:769] Sending connection context
0810 01:47:14.121247 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 01:47:14.121367 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":146,"threads_started":1}
I20260810 01:47:14.122340 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.113950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45816 (local address 127.9.77.189:38111)
0810 01:47:14.116451 (+ 2501us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:14.116457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:14.116473 (+ 16us) server_negotiation.cc:407] Connection header received
0810 01:47:14.116522 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:14.116526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:14.116617 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:14.117166 (+ 549us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:14.117319 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.118193 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.119943 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.120799 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.121723 (+ 924us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.121979 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:14.122029 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 01:47:14.122154 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2408,"thread_start_us":102,"threads_started":1}
I20260810 01:47:14.123044 19820 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 1 LEARNER]: Refusing update from remote peer 9ae815b0781f4d19aba6d526311d18c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 01:47:14.123216 19821 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260810 01:47:14.123960 19735 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: NON_VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:47:14.124141 19872 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ae815b0781f4d19aba6d526311d18c9. Latest consensus state: current_term: 1 leader_uuid: "9ae815b0781f4d19aba6d526311d18c9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: NON_VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: true } } }
I20260810 01:47:14.124270 19872 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [sys.catalog]: This master's current role is: LEARNER
I20260810 01:47:14.124744 19734 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: NON_VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 01:47:14.126070 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.055010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45812 (local address 127.9.77.189:38111)
0810 01:47:14.055179 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:14.055184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:14.055913 (+ 729us) server_negotiation.cc:407] Connection header received
0810 01:47:14.056095 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:14.056099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:14.056162 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:14.056255 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:14.056967 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.057805 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.111052 (+ 53247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.111502 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.111619 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.114319 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:14.114325 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:14.114333 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:14.114371 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:14.116451 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.117052 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.117056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.117059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.117167 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.117524 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.117528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.117530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.117774 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:14.117856 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:14.125639 (+ 7783us) server_negotiation.cc:299] Negotiation successful
0810 01:47:14.125851 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260810 01:47:14.126274 18874 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 -> Peer 646e25980e154c5c8b8c24f9040a9bb5 (127.9.77.188:45735): Couldn't send request to peer 646e25980e154c5c8b8c24f9040a9bb5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 01:47:14.126856 20026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.054894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:38111 (local address 127.0.0.1:45812)
0810 01:47:14.055820 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:14.055837 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:14.055962 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:14.056291 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:14.056293 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:14.056624 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:14.056824 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.056833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.109565 (+ 52732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.109573 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.110841 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.110857 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.111821 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.111828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.112119 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.112710 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:14.112725 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:14.114109 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:14.116606 (+ 2497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.116609 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.116611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.116924 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.117297 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.117301 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.117305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.117407 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.125194 (+ 7787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:14.125197 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:14.125373 (+ 176us) client_negotiation.cc:769] Sending connection context
0810 01:47:14.126275 (+ 902us) client_negotiation.cc:240] Negotiation successful
0810 01:47:14.126515 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"thread_start_us":118,"threads_started":1}
I20260810 01:47:14.126955 19821 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 646e25980e154c5c8b8c24f9040a9bb5 (127.9.77.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: NON_VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: true } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: NON_VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: true } } }
I20260810 01:47:14.130024 19739 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:47:14.130496 19739 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:47:14.130645 19739 tablet_replica.cc:330] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5: stopping tablet replica
I20260810 01:47:14.131417 19872 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ae815b0781f4d19aba6d526311d18c9" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: NON_VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: true } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: NON_VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: true } } }
I20260810 01:47:14.131531 19872 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [sys.catalog]: This master's current role is: LEARNER
I20260810 01:47:14.132558 19735 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9: attempting to promote NON_VOTER b3e994dc2a3d412ea8f13f43c39bc783 to VOTER
I20260810 01:47:14.133067 19735 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: NON_VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: true } }
W20260810 01:47:14.135419 18874 connection.cc:535] client connection to 127.9.77.188:45735 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 01:47:14.135525 18874 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 -> Peer 646e25980e154c5c8b8c24f9040a9bb5 (127.9.77.188:45735): Couldn't send request to peer 646e25980e154c5c8b8c24f9040a9bb5. Status: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20260810 01:47:14.136196 19820 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 1 LEARNER]: Refusing update from remote peer 9ae815b0781f4d19aba6d526311d18c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 01:47:14.137039 20043 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 01:47:14.139443 20044 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b3e994dc2a3d412ea8f13f43c39bc783 (127.9.77.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: NON_VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: true } } }
I20260810 01:47:14.140161 19820 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b3e994dc2a3d412ea8f13f43c39bc783 (127.9.77.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: NON_VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: true } } }
I20260810 01:47:14.141922 19734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9ae815b0781f4d19aba6d526311d18c9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: NON_VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: true } } }
I20260810 01:47:14.142045 19734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:14.142309 19734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ae815b0781f4d19aba6d526311d18c9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: NON_VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: true } } }
I20260810 01:47:14.142410 19734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:14.142584 19872 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ae815b0781f4d19aba6d526311d18c9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: NON_VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: true } } }
I20260810 01:47:14.142690 19872 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 01:47:14.149034 19739 master.cc:552] Master@127.9.77.188:45735 shutdown complete.
I20260810 01:47:14.149106 19739 master_runner.cc:305] Clearing existing system tablet
I20260810 01:47:14.149415 19739 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:14.149479 19739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:14.151010 19739 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 01:47:14.152107 20048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:14.152380 19739 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:14.152658 19739 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 01:47:14.152742 19739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-2/wal
uuid: "646e25980e154c5c8b8c24f9040a9bb5"
format_stamp: "Formatted at 2026-08-10 01:47:13 on dist-test-slave-kvfs"
server_key: "2678b268cdba301751837f2936516c77"
server_key_iv: "fb8d6d88e4bb332baebf54af255e4336"
server_key_version: "encryptionkey@0"
I20260810 01:47:14.155676 19739 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:47:14.158571 19739 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 01:47:14.158754 19739 log.cc:1196] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 01:47:14.159202 19739 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5: Deleting consensus metadata
I20260810 01:47:14.159583 19739 master_runner.cc:315] Copying system tablet from 127.9.77.190:33329
I20260810 01:47:14.161116 19739 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5: tablet copy: Beginning tablet copy session from remote peer at address 127.9.77.190:33329
I20260810 01:47:14.172447 20057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.161925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.0.0.1:36598)
0810 01:47:14.162336 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:14.162354 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:14.162508 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:14.162883 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:14.162887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:14.163168 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:14.163355 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.163364 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.164590 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.164594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.165703 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.165717 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.166359 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.166363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.166478 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.167260 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:14.167279 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:14.167846 (+ 567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:14.170215 (+ 2369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.170219 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.170222 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.170567 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.170965 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.170969 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.170971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.171087 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.171675 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:14.171679 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:14.171809 (+ 130us) client_negotiation.cc:769] Sending connection context
0810 01:47:14.171935 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 01:47:14.172164 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":116,"threads_started":1}
I20260810 01:47:14.172492 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.162068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36598 (local address 127.9.77.190:33329)
0810 01:47:14.162228 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:14.162233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:14.162461 (+ 228us) server_negotiation.cc:407] Connection header received
0810 01:47:14.162656 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:14.162659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:14.162727 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:14.162842 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:14.163503 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.164436 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.165844 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.166212 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.166339 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.168060 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:14.168065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:14.168068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:14.168100 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:14.170086 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.170746 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.170751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.170753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.170853 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.171259 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.171262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.171264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.171559 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:14.171667 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:14.172006 (+ 339us) server_negotiation.cc:299] Negotiation successful
0810 01:47:14.172209 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 01:47:14.173421 18922 tablet_copy_service.cc:140] P 9ae815b0781f4d19aba6d526311d18c9: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 646e25980e154c5c8b8c24f9040a9bb5 ({username='oryx', principal='oryx/127.9.77.188@KRBTEST.COM'} at 127.0.0.1:36598)
I20260810 01:47:14.173559 18922 tablet_copy_service.cc:161] P 9ae815b0781f4d19aba6d526311d18c9: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 646e25980e154c5c8b8c24f9040a9bb5 at {username='oryx', principal='oryx/127.9.77.188@KRBTEST.COM'} at 127.0.0.1:36598: session id = 646e25980e154c5c8b8c24f9040a9bb5-00000000000000000000000000000000
I20260810 01:47:14.174976 18922 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 01:47:14.176724 19739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 01:47:14.179716 19739 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5: tablet copy: Starting download of 0 data blocks...
I20260810 01:47:14.179960 19739 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5: tablet copy: Starting download of 1 WAL segments...
I20260810 01:47:14.182045 19739 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 01:47:14.187160 18922 tablet_copy_service.cc:342] P 9ae815b0781f4d19aba6d526311d18c9: Request end of tablet copy session 646e25980e154c5c8b8c24f9040a9bb5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.77.188@KRBTEST.COM'} at 127.0.0.1:36598
I20260810 01:47:14.187340 18922 tablet_copy_service.cc:433] P 9ae815b0781f4d19aba6d526311d18c9: ending tablet copy session 646e25980e154c5c8b8c24f9040a9bb5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 646e25980e154c5c8b8c24f9040a9bb5
I20260810 01:47:14.196960 19739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:14.199368 20058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:14.199661 20059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:14.199666 20065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:14.200245 19739 server_base.cc:1034] running on GCE node
I20260810 01:47:14.200464 19739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:14.200974 19739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:14.202176 19739 hybrid_clock.cc:648] HybridClock initialized: now 1786326434202155 us; error 49 us; skew 500 ppm
Aug 10 01:47:14 dist-test-slave-kvfs krb5kdc[18817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326434, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:14.204279 19739 init.cc:377] Logged in from keytab as oryx/127.9.77.188@KRBTEST.COM (short username oryx)
I20260810 01:47:14.204869 19739 webserver.cc:460] Webserver started at http://127.9.77.188:38901/ using document root <none> and password file <none>
I20260810 01:47:14.205155 19739 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:14.205247 19739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:14.206805 19739 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 01:47:14.207962 20073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:14.208384 19739 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 01:47:14.208509 19739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-2/wal
uuid: "646e25980e154c5c8b8c24f9040a9bb5"
format_stamp: "Formatted at 2026-08-10 01:47:13 on dist-test-slave-kvfs"
server_key: "2678b268cdba301751837f2936516c77"
server_key_iv: "fb8d6d88e4bb332baebf54af255e4336"
server_key_version: "encryptionkey@0"
I20260810 01:47:14.208688 19739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:14.217104 19739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:14.219575 19739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:14.225302 19739 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.188:45735
I20260810 01:47:14.225332 20150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.188:45735 every 8 connection(s)
I20260810 01:47:14.226239 19739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/master-2/data/info.pb
I20260810 01:47:14.228927 20151 sys_catalog.cc:263] Verifying existing consensus state
I20260810 01:47:14.230505 20151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5: Bootstrap starting.
I20260810 01:47:14.245280 20151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 01:47:14.245733 20151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5: Bootstrap complete.
I20260810 01:47:14.246692 20151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: NON_VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: true } }
I20260810 01:47:14.246910 20151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 646e25980e154c5c8b8c24f9040a9bb5, State: Initialized, Role: LEARNER
I20260810 01:47:14.247080 20151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: NON_VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: true } }
I20260810 01:47:14.247902 20151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:47:14.249330 20155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: NON_VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: true } } }
I20260810 01:47:14.249529 20155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [sys.catalog]: This master's current role is: LEARNER
I20260810 01:47:14.255996 20170 catalog_manager.cc:1259] Loaded cluster ID: 30255652128446de977d487ec2e58d4b
I20260810 01:47:14.256108 20170 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5: loading cluster ID for follower catalog manager: success
I20260810 01:47:14.259367 20170 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5: acquiring CA information for follower catalog manager: success
I20260810 01:47:14.261734 20170 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 01:47:14.309296 19572 heartbeater.cc:503] Master 127.9.77.190:33329 was elected leader, sending a full tablet report...
W20260810 01:47:14.321017 20040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:14.321473 20040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:14.321565 20040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:14.326224 20040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:14.326339 20040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:14.326447 20040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:47:14.332937 20040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.129.keytab
--principal=oryx/127.9.77.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:45485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=37359
--tserver_master_addrs=127.9.77.190:33329,127.9.77.189:38111,127.9.77.188:45735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:14.334509 20040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:14.335887 20040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:14.348380 20190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:14.348378 20182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:14.348378 20184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:14.348719 20040 server_base.cc:1034] running on GCE node
I20260810 01:47:14.349228 20040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:14.350087 20040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:14.351313 20040 hybrid_clock.cc:648] HybridClock initialized: now 1786326434351286 us; error 33 us; skew 500 ppm
Aug 10 01:47:14 dist-test-slave-kvfs krb5kdc[18817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326434, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:14.354734 20040 init.cc:377] Logged in from keytab as oryx/127.9.77.129@KRBTEST.COM (short username oryx)
I20260810 01:47:14.355773 20040 webserver.cc:460] Webserver started at http://127.9.77.129:37359/ using document root <none> and password file <none>
I20260810 01:47:14.356515 20040 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:14.356604 20040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:14.361568 20040 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260810 01:47:14.364197 20200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:14.365789 20040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 01:47:14.365948 20040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "3ad0891514f242b8a78735404fbbcc19"
format_stamp: "Formatted at 2026-08-10 01:47:12 on dist-test-slave-kvfs"
server_key: "2e6c284eeb9f59cd42a4e889877f837c"
server_key_iv: "a712316147ddbe69f2211ef1012932bc"
server_key_version: "encryptionkey@0"
I20260810 01:47:14.366647 20040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:14.389379 20040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:14.394088 20040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:14.394353 20040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:14.395183 20040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:14.396412 20040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:14.396469 20040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:14.396546 20040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:14.396577 20040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:14.408964 20040 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:45485
I20260810 01:47:14.408993 20373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:45485 every 8 connection(s)
I20260810 01:47:14.410449 20040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:47:14.414927 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 20040
I20260810 01:47:14.415185 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 19181
I20260810 01:47:14.451378 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:46775
--local_ip_for_outbound_sockets=127.9.77.130
--tserver_master_addrs=127.9.77.190:33329
--webserver_port=35879
--webserver_interface=127.9.77.130
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.130.keytab
--principal=oryx/127.9.77.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.77.190:33329,127.9.77.189:38111
--tserver_master_addrs=127.9.77.190:33329,127.9.77.189:38111,127.9.77.188:45735 with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 01:47:14 dist-test-slave-kvfs krb5kdc[18817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326434, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.189@KRBTEST.COM
Aug 10 01:47:14 dist-test-slave-kvfs krb5kdc[18817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326434, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM
Aug 10 01:47:14 dist-test-slave-kvfs krb5kdc[18817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326434, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.188@KRBTEST.COM
I20260810 01:47:14.459156 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.412889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:49443 (local address 127.9.77.189:38111)
0810 01:47:14.413082 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:14.413088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:14.442677 (+ 29589us) server_negotiation.cc:407] Connection header received
0810 01:47:14.442887 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:14.442892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:14.442967 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:14.443052 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:14.445716 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.446641 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.449075 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.449456 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.449534 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.454453 (+ 4919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:14.454456 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:14.454459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:14.454490 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:14.456715 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.457373 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.457380 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.457385 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.457485 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.457833 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.457839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.457843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.458111 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:14.458201 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:14.458688 (+ 487us) server_negotiation.cc:299] Negotiation successful
0810 01:47:14.458859 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 01:47:14.459178 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.414800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:33411 (local address 127.9.77.190:33329)
0810 01:47:14.414946 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:14.414955 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:14.443559 (+ 28604us) server_negotiation.cc:407] Connection header received
0810 01:47:14.443705 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:14.443709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:14.443780 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:14.443905 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:14.445717 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.446633 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.449075 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.449444 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.449534 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.454001 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:14.454008 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:14.454012 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:14.454055 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:14.456779 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.457736 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.457739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.457741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.457848 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.458151 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.458154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.458156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.458411 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:14.458505 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:14.458847 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 01:47:14.459004 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 01:47:14.460496 20389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.413481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:45735 (local address 127.9.77.129:56175)
0810 01:47:14.414165 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:14.414225 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:14.442390 (+ 28165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:14.443606 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:14.443623 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:14.444434 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:14.445508 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.445534 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.447099 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.447105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.448934 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.448995 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.450000 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.450006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.450164 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.450970 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:14.450995 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:14.454025 (+ 3030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:14.456502 (+ 2477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.456509 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.456532 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.456834 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.457171 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.457174 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.457176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.457355 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.457946 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:14.457955 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:14.458361 (+ 406us) client_negotiation.cc:769] Sending connection context
0810 01:47:14.458597 (+ 236us) client_negotiation.cc:240] Negotiation successful
0810 01:47:14.459035 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":671,"spinlock_wait_cycles":34560,"thread_start_us":191,"threads_started":1}
I20260810 01:47:14.460497 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.413481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:38111 (local address 127.9.77.129:49443)
0810 01:47:14.442469 (+ 28988us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:14.442493 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:14.442734 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:14.443707 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:14.443709 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:14.444635 (+ 926us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:14.445508 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.445533 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.446852 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.446855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.448938 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.448944 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.449801 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.449805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.450384 (+ 579us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.451504 (+ 1120us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:14.451519 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:14.454275 (+ 2756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:14.456908 (+ 2633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.456914 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.456919 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.457235 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.457604 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.457610 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.457614 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.457711 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.458266 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:14.458271 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:14.458393 (+ 122us) client_negotiation.cc:769] Sending connection context
0810 01:47:14.459454 (+ 1061us) client_negotiation.cc:240] Negotiation successful
0810 01:47:14.459641 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28825,"mutex_wait_us":1028,"spinlock_wait_cycles":12544,"thread_start_us":180,"threads_started":1}
I20260810 01:47:14.461447 20386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.413737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.9.77.129:33411)
0810 01:47:14.443436 (+ 29699us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:14.443449 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:14.443580 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:14.444006 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:14.444011 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:14.444423 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:14.445508 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.445533 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.446852 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.446855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.448913 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.448925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.449813 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.449818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.450378 (+ 560us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.451107 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:14.451130 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:14.453732 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:14.456975 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.456978 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.456980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.457613 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.457958 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.457960 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.457962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.458046 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.458565 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:14.458571 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:14.458654 (+ 83us) client_negotiation.cc:769] Sending connection context
0810 01:47:14.459847 (+ 1193us) client_negotiation.cc:240] Negotiation successful
0810 01:47:14.460001 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29613,"mutex_wait_us":676,"spinlock_wait_cycles":4992,"thread_start_us":117,"threads_started":1}
I20260810 01:47:14.461606 20385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.412811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:56175 (local address 127.9.77.188:45735)
0810 01:47:14.413146 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:14.413157 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:14.414347 (+ 1190us) server_negotiation.cc:407] Connection header received
0810 01:47:14.442951 (+ 28604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:14.442959 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:14.443054 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:14.443133 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:14.445731 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.446926 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.449134 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.449590 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.449701 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.454615 (+ 4914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:14.454622 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:14.454626 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:14.454669 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:14.456371 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.456955 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.456960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.456964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.457074 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.457468 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.457471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.457473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.457774 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:14.457880 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:14.461189 (+ 3309us) server_negotiation.cc:299] Negotiation successful
0810 01:47:14.461371 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":122,"threads_started":1}
I20260810 01:47:14.462327 20374 heartbeater.cc:347] Connected to a master server at 127.9.77.188:45735
I20260810 01:47:14.462539 20377 heartbeater.cc:347] Connected to a master server at 127.9.77.190:33329
I20260810 01:47:14.462361 20379 heartbeater.cc:347] Connected to a master server at 127.9.77.189:38111
I20260810 01:47:14.462638 20374 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:14.462644 20377 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:14.462694 20379 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:14.463421 20377 heartbeater.cc:511] Master 127.9.77.190:33329 requested a full tablet report, sending...
I20260810 01:47:14.463423 20374 heartbeater.cc:511] Master 127.9.77.188:45735 requested a full tablet report, sending...
I20260810 01:47:14.463423 20379 heartbeater.cc:511] Master 127.9.77.189:38111 requested a full tablet report, sending...
I20260810 01:47:14.465356 18891 ts_manager.cc:194] Re-registered known tserver with Master: 3ad0891514f242b8a78735404fbbcc19 (127.9.77.129:45485)
I20260810 01:47:14.465793 20096 ts_manager.cc:194] Registered new tserver with Master: 3ad0891514f242b8a78735404fbbcc19 (127.9.77.129:45485)
I20260810 01:47:14.466284 18891 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.77.129@KRBTEST.COM'} at 127.9.77.129:33411
I20260810 01:47:14.467587 19803 ts_manager.cc:194] Registered new tserver with Master: 3ad0891514f242b8a78735404fbbcc19 (127.9.77.129:45485)
W20260810 01:47:14.595060 20394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:14.595448 20394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:14.595533 20394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:14.599699 20394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:14.599819 20394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:14.599927 20394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:47:14.606007 20394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.130.keytab
--principal=oryx/127.9.77.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:46775
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=35879
--tserver_master_addrs=127.9.77.190:33329,127.9.77.189:38111,127.9.77.188:45735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:14.607493 20394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:14.608968 20394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:14.620963 20403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:14.620994 20404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:14.621188 20394 server_base.cc:1034] running on GCE node
W20260810 01:47:14.620990 20408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:14.621948 20394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:14.622838 20394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:14.624118 20394 hybrid_clock.cc:648] HybridClock initialized: now 1786326434624088 us; error 37 us; skew 500 ppm
Aug 10 01:47:14 dist-test-slave-kvfs krb5kdc[18817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326434, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:14.627488 20394 init.cc:377] Logged in from keytab as oryx/127.9.77.130@KRBTEST.COM (short username oryx)
I20260810 01:47:14.628613 20394 webserver.cc:460] Webserver started at http://127.9.77.130:35879/ using document root <none> and password file <none>
I20260810 01:47:14.629369 20394 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:14.629454 20394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:14.634485 20394 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 01:47:14.637264 20419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:14.638994 20394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:47:14.639140 20394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "12e99ab08944444b8ea106fa5aa38769"
format_stamp: "Formatted at 2026-08-10 01:47:12 on dist-test-slave-kvfs"
server_key: "568941a6858db70fc1e05205d1249742"
server_key_iv: "d0ed527293182d7ecc4550cda1f34bea"
server_key_version: "encryptionkey@0"
I20260810 01:47:14.639827 20394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:14.659189 20394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:14.661791 20385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.657584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51066 (local address 127.9.77.188:45735)
0810 01:47:14.657782 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:14.657788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:14.657935 (+ 147us) server_negotiation.cc:407] Connection header received
0810 01:47:14.658167 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:14.658170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:14.658243 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:14.658378 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:14.658697 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.659268 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.660643 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.661159 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.661290 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.661509 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:14.661547 (+ 38us) server_negotiation.cc:299] Negotiation successful
0810 01:47:14.661637 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260810 01:47:14.661798 20429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.657457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:45735 (local address 127.0.0.1:51066)
0810 01:47:14.657789 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:14.657807 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:14.658024 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:14.658398 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:14.658402 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:14.658424 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:14.658568 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.658575 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.659398 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.659404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.660501 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.660513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.661306 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.661310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.661410 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.661438 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 01:47:14.661509 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 01:47:14.661615 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":108,"threads_started":1}
I20260810 01:47:14.663059 20109 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 01:47:14.663582 20155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ae815b0781f4d19aba6d526311d18c9. Latest consensus state: current_term: 1 leader_uuid: "9ae815b0781f4d19aba6d526311d18c9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: NON_VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: true } } }
I20260810 01:47:14.663791 20155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [sys.catalog]: This master's current role is: LEARNER
I20260810 01:47:14.663849 20394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:14.664103 20394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:14.664929 20394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:14.666055 20394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:14.666136 20394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:14.666217 20394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:14.666273 20394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:14.677826 20394 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:46775
I20260810 01:47:14.677850 20595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:46775 every 8 connection(s)
I20260810 01:47:14.679144 20394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:47:14.684445 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 20394
I20260810 01:47:14.684700 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 19377
Aug 10 01:47:14 dist-test-slave-kvfs krb5kdc[18817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326434, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.188@KRBTEST.COM
Aug 10 01:47:14 dist-test-slave-kvfs krb5kdc[18817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326434, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM
I20260810 01:47:14.710531 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.681602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:34511 (local address 127.9.77.190:33329)
0810 01:47:14.681794 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:14.681800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:14.683456 (+ 1656us) server_negotiation.cc:407] Connection header received
0810 01:47:14.698818 (+ 15362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:14.698823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:14.698914 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:14.698979 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:14.699941 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.700712 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.701823 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.702160 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.702250 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.705274 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:14.705279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:14.705282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:14.705316 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:14.707074 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.707910 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.707914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.707916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.708051 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.708512 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.708516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.708518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.708861 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:14.708985 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:14.709996 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0810 01:47:14.710223 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 01:47:14.711762 20385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.682705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:52667 (local address 127.9.77.188:45735)
0810 01:47:14.682882 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:14.682888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:14.684484 (+ 1596us) server_negotiation.cc:407] Connection header received
0810 01:47:14.690819 (+ 6335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:14.690827 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:14.690940 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:14.691031 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:14.694584 (+ 3553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.695402 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.697925 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.698439 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.698563 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.704264 (+ 5701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:14.704273 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:14.704279 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:14.704327 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:14.706963 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.709714 (+ 2751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.709721 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.709726 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.709860 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.710292 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.710299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.710303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.710663 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:14.710776 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:14.711278 (+ 502us) server_negotiation.cc:299] Negotiation successful
0810 01:47:14.711485 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 01:47:14.713693 20602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.682514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:45735 (local address 127.9.77.130:52667)
0810 01:47:14.684328 (+ 1814us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:14.684346 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:14.684751 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:14.691853 (+ 7102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:14.691872 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:14.693057 (+ 1185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:14.694304 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.694338 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.695584 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.695590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.697710 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.697730 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.698926 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.698936 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.699274 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.700478 (+ 1204us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:14.700514 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:14.703950 (+ 3436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:14.707157 (+ 3207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.707169 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.707201 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.709557 (+ 2356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.710010 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.710016 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.710021 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.710154 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.710858 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:14.710861 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:14.711000 (+ 139us) client_negotiation.cc:769] Sending connection context
0810 01:47:14.712068 (+ 1068us) client_negotiation.cc:240] Negotiation successful
0810 01:47:14.712254 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1731,"thread_start_us":159,"threads_started":1}
I20260810 01:47:14.713701 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.682193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.9.77.130:34511)
0810 01:47:14.683230 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:14.683302 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:14.698590 (+ 15288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:14.699170 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:14.699174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:14.699523 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:14.699755 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.699765 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.700933 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.700937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.701672 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.701681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.702475 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.702479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.702573 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.703428 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:14.703445 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:14.705075 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:14.707226 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.707230 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.707232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.707777 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.708175 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.708178 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.708180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.708397 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.709170 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:14.709182 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:14.709648 (+ 466us) client_negotiation.cc:769] Sending connection context
0810 01:47:14.711293 (+ 1645us) client_negotiation.cc:240] Negotiation successful
0810 01:47:14.711793 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"thread_start_us":183,"threads_started":1}
I20260810 01:47:14.715471 20597 heartbeater.cc:347] Connected to a master server at 127.9.77.190:33329
I20260810 01:47:14.715827 20597 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:14.715466 20596 heartbeater.cc:347] Connected to a master server at 127.9.77.188:45735
I20260810 01:47:14.716413 20596 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:14.716694 20597 heartbeater.cc:511] Master 127.9.77.190:33329 requested a full tablet report, sending...
I20260810 01:47:14.716710 20596 heartbeater.cc:511] Master 127.9.77.188:45735 requested a full tablet report, sending...
I20260810 01:47:14.718297 18891 ts_manager.cc:194] Re-registered known tserver with Master: 12e99ab08944444b8ea106fa5aa38769 (127.9.77.130:46775)
I20260810 01:47:14.718384 20096 ts_manager.cc:194] Registered new tserver with Master: 12e99ab08944444b8ea106fa5aa38769 (127.9.77.130:46775)
I20260810 01:47:14.719820 18891 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.77.130@KRBTEST.COM'} at 127.9.77.130:34511
Aug 10 01:47:14 dist-test-slave-kvfs krb5kdc[18817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326434, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.189@KRBTEST.COM
I20260810 01:47:14.723377 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:39543
--local_ip_for_outbound_sockets=127.9.77.131
--tserver_master_addrs=127.9.77.190:33329
--webserver_port=34353
--webserver_interface=127.9.77.131
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.131.keytab
--principal=oryx/127.9.77.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.77.190:33329,127.9.77.189:38111
--tserver_master_addrs=127.9.77.190:33329,127.9.77.189:38111,127.9.77.188:45735 with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 01:47:14.725543 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.682193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:38111 (local address 127.9.77.130:42747)
0810 01:47:14.683634 (+ 1441us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:14.683651 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:14.684743 (+ 1092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:14.713242 (+ 28499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:14.713247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:14.713510 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:14.713669 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.713679 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.715064 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.715068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.716112 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.716126 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.717136 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.717141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.717271 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.718940 (+ 1669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:14.718960 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:14.721011 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:14.723668 (+ 2657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.723672 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.723676 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.723917 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.724364 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.724367 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.724368 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.724451 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.724983 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:14.724985 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:14.725076 (+ 91us) client_negotiation.cc:769] Sending connection context
0810 01:47:14.725156 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 01:47:14.725355 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1340,"mutex_wait_us":64,"spinlock_wait_cycles":68352,"thread_start_us":192,"threads_started":1}
I20260810 01:47:14.725723 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.681599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:42747 (local address 127.9.77.189:38111)
0810 01:47:14.681804 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:14.681810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:14.683810 (+ 2000us) server_negotiation.cc:407] Connection header received
0810 01:47:14.712872 (+ 29062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:14.712878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:14.712966 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:14.713038 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:14.713911 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.714870 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.716298 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.716681 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.716834 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.721227 (+ 4393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:14.721232 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:14.721236 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:14.721284 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:14.723492 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.724115 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.724123 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.724128 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.724249 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.724562 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.724569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.724573 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.724884 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:14.725000 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:14.725264 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 01:47:14.725459 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 01:47:14.726472 20599 heartbeater.cc:347] Connected to a master server at 127.9.77.189:38111
I20260810 01:47:14.726644 20599 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:14.726955 20599 heartbeater.cc:511] Master 127.9.77.189:38111 requested a full tablet report, sending...
I20260810 01:47:14.728045 19803 ts_manager.cc:194] Registered new tserver with Master: 12e99ab08944444b8ea106fa5aa38769 (127.9.77.130:46775)
W20260810 01:47:14.874562 20604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:14.874982 20604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:14.875069 20604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:14.879350 20604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:14.879467 20604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:14.879570 20604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:47:14.886471 20604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.131.keytab
--principal=oryx/127.9.77.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:39543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=34353
--tserver_master_addrs=127.9.77.190:33329,127.9.77.189:38111,127.9.77.188:45735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:14.888176 20604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:14.889603 20604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:14.901526 20612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:14.901535 20611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:14.901527 20616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:14.901880 20604 server_base.cc:1034] running on GCE node
I20260810 01:47:14.902503 20604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:14.903276 20604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:14.904472 20604 hybrid_clock.cc:648] HybridClock initialized: now 1786326434904433 us; error 49 us; skew 500 ppm
Aug 10 01:47:14 dist-test-slave-kvfs krb5kdc[18817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326434, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:14.907702 20604 init.cc:377] Logged in from keytab as oryx/127.9.77.131@KRBTEST.COM (short username oryx)
I20260810 01:47:14.908824 20604 webserver.cc:460] Webserver started at http://127.9.77.131:34353/ using document root <none> and password file <none>
I20260810 01:47:14.909603 20604 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:14.909693 20604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:14.914530 20604 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260810 01:47:14.917032 20625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:14.918617 20604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 01:47:14.918768 20604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "49e30d0b090648f29deeb808001b14a7"
format_stamp: "Formatted at 2026-08-10 01:47:13 on dist-test-slave-kvfs"
server_key: "9c9075da294d5e67e00b5638439df55a"
server_key_iv: "e9375b1d7084c5cb02206d622cf0344d"
server_key_version: "encryptionkey@0"
I20260810 01:47:14.919448 20604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:14.934559 20604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:14.938181 20604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:14.938423 20604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:14.939275 20604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:14.940382 20604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:14.940433 20604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:14.940511 20604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:14.940544 20604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:14.952682 20604 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:39543
I20260810 01:47:14.952766 20805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:39543 every 8 connection(s)
I20260810 01:47:14.954038 20604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:47:14.955325 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 20604
Aug 10 01:47:14 dist-test-slave-kvfs krb5kdc[18817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326434, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.188@KRBTEST.COM
Aug 10 01:47:14 dist-test-slave-kvfs krb5kdc[18817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326434, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM
I20260810 01:47:14.975659 20825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.958627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51082 (local address 127.9.77.188:45735)
0810 01:47:14.961078 (+ 2451us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:14.961083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:14.961097 (+ 14us) server_negotiation.cc:407] Connection header received
0810 01:47:14.961130 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:14.961133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:14.961179 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:14.961279 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:14.962597 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.963664 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.965305 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.965696 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.966121 (+ 425us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.968053 (+ 1932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:14.968058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:14.968061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:14.968093 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:14.972763 (+ 4670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.973335 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.973338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.973340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.973436 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.973759 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.973763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.973765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.974033 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:14.975126 (+ 1093us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:14.975210 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 01:47:14.975398 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2371,"spinlock_wait_cycles":5376,"thread_start_us":130,"threads_started":1}
I20260810 01:47:14.977174 20385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.956725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:43365 (local address 127.9.77.188:45735)
0810 01:47:14.957167 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:14.957173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:14.960303 (+ 3130us) server_negotiation.cc:407] Connection header received
0810 01:47:14.960990 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:14.960994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:14.961057 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:14.961139 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:14.963377 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.964193 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.966052 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.966424 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.966514 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.970656 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:14.970663 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:14.970666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:14.970702 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:14.973696 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.975002 (+ 1306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.975009 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.975013 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.975116 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.975647 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.975651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.975653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.975853 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:14.975916 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:14.976837 (+ 921us) server_negotiation.cc:299] Negotiation successful
0810 01:47:14.976969 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":312,"spinlock_wait_cycles":1408}
I20260810 01:47:14.977802 20823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.958040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36602 (local address 127.9.77.190:33329)
0810 01:47:14.958376 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:14.958384 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:14.961151 (+ 2767us) server_negotiation.cc:407] Connection header received
0810 01:47:14.961190 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:14.961193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:14.961237 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:14.961333 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:14.963614 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.965726 (+ 2112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.967647 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.968151 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.968258 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.970801 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:14.970805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:14.970808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:14.970852 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:14.974241 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.975430 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.975433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.975436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.975540 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.976112 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.976116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.976118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.976363 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:14.976501 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:14.977447 (+ 946us) server_negotiation.cc:299] Negotiation successful
0810 01:47:14.977600 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":138,"threads_started":1}
I20260810 01:47:14.979069 20821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.957254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:45735 (local address 127.9.77.131:43365)
0810 01:47:14.960183 (+ 2929us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:14.960202 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:14.960582 (+ 380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:14.961484 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:14.961495 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:14.962279 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:14.963173 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.963200 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.964357 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.964361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.965907 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.965922 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.966708 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.966712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.966948 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.967587 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:14.967607 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:14.970429 (+ 2822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:14.973825 (+ 3396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.973832 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.973856 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.974878 (+ 1022us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.975261 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.975268 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.975273 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.975509 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.976007 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:14.976014 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:14.976514 (+ 500us) client_negotiation.cc:769] Sending connection context
0810 01:47:14.976794 (+ 280us) client_negotiation.cc:240] Negotiation successful
0810 01:47:14.977538 (+ 744us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":21,"thread_start_us":156,"threads_started":1}
I20260810 01:47:14.979475 20829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.959569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45826 (local address 127.9.77.189:38111)
0810 01:47:14.959848 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:14.959855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:14.959872 (+ 17us) server_negotiation.cc:407] Connection header received
0810 01:47:14.959945 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:14.959953 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:14.960059 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:14.962176 (+ 2117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:14.962321 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.963533 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.968172 (+ 4639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.968515 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.969058 (+ 543us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.970398 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:14.970401 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:14.970403 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:14.970427 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:14.973241 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.975541 (+ 2300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.975544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.975546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.975632 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.976596 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.976599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.976601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.978207 (+ 1606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:14.979051 (+ 844us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:14.979153 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 01:47:14.979305 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":102,"threads_started":1}
Aug 10 01:47:14 dist-test-slave-kvfs krb5kdc[18817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326434, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.189@KRBTEST.COM
I20260810 01:47:14.980803 20808 heartbeater.cc:347] Connected to a master server at 127.9.77.188:45735
I20260810 01:47:14.981164 20808 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:14.981961 20808 heartbeater.cc:511] Master 127.9.77.188:45735 requested a full tablet report, sending...
I20260810 01:47:14.983498 20096 ts_manager.cc:194] Registered new tserver with Master: 49e30d0b090648f29deeb808001b14a7 (127.9.77.131:39543)
I20260810 01:47:14.983882 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.958892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:60047 (local address 127.9.77.189:38111)
0810 01:47:14.959059 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:14.959064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:14.960693 (+ 1629us) server_negotiation.cc:407] Connection header received
0810 01:47:14.960909 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:14.960913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:14.960971 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:14.961035 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:14.968620 (+ 7585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.969457 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.977046 (+ 7589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.977446 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.977526 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.980472 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:14.980476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:14.980478 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:14.980504 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:14.982120 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.982561 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.982564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.982566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.982674 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.983007 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.983011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.983012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.983225 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:14.983295 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:14.983605 (+ 310us) server_negotiation.cc:299] Negotiation successful
0810 01:47:14.983727 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 01:47:14.984310 20822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.958685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:38111 (local address 127.9.77.131:60047)
0810 01:47:14.960580 (+ 1895us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:14.960596 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:14.960774 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:14.961487 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:14.961496 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:14.967460 (+ 5964us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:14.968469 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.968482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.973350 (+ 4868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.973353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.974417 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.974427 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.977702 (+ 3275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.977705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.977783 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.978289 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:14.978309 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:14.980321 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:14.982233 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.982236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.982246 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.982451 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.982792 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.982796 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.982797 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.982898 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.983345 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:14.983347 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:14.983420 (+ 73us) client_negotiation.cc:769] Sending connection context
0810 01:47:14.983536 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 01:47:14.983694 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1818,"thread_start_us":110,"threads_started":1}
I20260810 01:47:14.984951 20811 heartbeater.cc:347] Connected to a master server at 127.9.77.189:38111
I20260810 01:47:14.985067 20811 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:14.985265 20811 heartbeater.cc:511] Master 127.9.77.189:38111 requested a full tablet report, sending...
I20260810 01:47:14.986044 19803 ts_manager.cc:194] Registered new tserver with Master: 49e30d0b090648f29deeb808001b14a7 (127.9.77.131:39543)
I20260810 01:47:14.982276 18891 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 01:47:14.987396 18891 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:47:14.991460 20043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ae815b0781f4d19aba6d526311d18c9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: NON_VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: true } } }
I20260810 01:47:14.991631 20043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:14.993120 20826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.957254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.9.77.131:49735)
0810 01:47:14.959206 (+ 1952us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:14.959290 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:14.960582 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:14.961484 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:14.961495 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:14.963753 (+ 2258us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:14.963888 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.963895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.965065 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.965069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.967362 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:14.968592 (+ 1230us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.972114 (+ 3522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.972120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:14.972233 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.972800 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:14.972821 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:14.982966 (+ 10145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:14.990269 (+ 7303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.990272 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.990274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.990557 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.991657 (+ 1100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:14.991660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:14.991662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:14.991837 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:14.992483 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:14.992486 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:14.992601 (+ 115us) client_negotiation.cc:769] Sending connection context
0810 01:47:14.992710 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 01:47:14.992914 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1691,"mutex_wait_us":32,"spinlock_wait_cycles":43904,"thread_start_us":196,"threads_started":1}
I20260810 01:47:14.993156 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:14.956964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:49735 (local address 127.9.77.190:33329)
0810 01:47:14.957311 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:14.957316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:14.959436 (+ 2120us) server_negotiation.cc:407] Connection header received
0810 01:47:14.960846 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:14.960853 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:14.960923 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:14.961030 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:14.964001 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.964944 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.969620 (+ 4676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:14.970091 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:14.970210 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:14.988188 (+ 17978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:14.988193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:14.988196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:14.988233 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:14.990111 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.990697 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.990700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.990702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.990804 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:14.991982 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:14.991989 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:14.991992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:14.992356 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:14.992482 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:14.992840 (+ 358us) server_negotiation.cc:299] Negotiation successful
0810 01:47:14.992993 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235}
I20260810 01:47:14.994091 20809 heartbeater.cc:347] Connected to a master server at 127.9.77.190:33329
I20260810 01:47:14.994223 20809 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:14.994484 20809 heartbeater.cc:511] Master 127.9.77.190:33329 requested a full tablet report, sending...
I20260810 01:47:14.995321 18889 ts_manager.cc:194] Re-registered known tserver with Master: 49e30d0b090648f29deeb808001b14a7 (127.9.77.131:39543)
I20260810 01:47:14.996238 18889 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.77.131@KRBTEST.COM'} at 127.9.77.131:49735
I20260810 01:47:14.996938 20840 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9: attempting to promote NON_VOTER 646e25980e154c5c8b8c24f9040a9bb5 to VOTER
I20260810 01:47:14.997362 20840 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: false } }
I20260810 01:47:15.000568 19820 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 1 FOLLOWER]: Refusing update from remote peer 9ae815b0781f4d19aba6d526311d18c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 01:47:15.001312 20108 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [term 1 LEARNER]: Refusing update from remote peer 9ae815b0781f4d19aba6d526311d18c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 01:47:15.001585 20043 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 01:47:15.002514 20043 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 01:47:15.003844 20043 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 646e25980e154c5c8b8c24f9040a9bb5 (127.9.77.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: false } } }
I20260810 01:47:15.004558 19820 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 646e25980e154c5c8b8c24f9040a9bb5 (127.9.77.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: false } } }
I20260810 01:47:15.005630 20108 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 646e25980e154c5c8b8c24f9040a9bb5 (127.9.77.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: false } } }
I20260810 01:47:15.006471 20839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9ae815b0781f4d19aba6d526311d18c9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: false } } }
I20260810 01:47:15.006608 20839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:15.006839 20043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ae815b0781f4d19aba6d526311d18c9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: false } } }
I20260810 01:47:15.006965 20043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:15.007092 20839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ae815b0781f4d19aba6d526311d18c9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: false } } }
I20260810 01:47:15.007194 20839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9 [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:15.007473 20853 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ae815b0781f4d19aba6d526311d18c9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: false } } }
I20260810 01:47:15.007632 20853 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 01:47:15.007937 20155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ae815b0781f4d19aba6d526311d18c9" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: false } } }
I20260810 01:47:15.008101 20155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 01:47:15.015586 20429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:15.008836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:39543 (local address 127.0.0.1:43208)
0810 01:47:15.009135 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:15.009155 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:15.009272 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:15.010841 (+ 1569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:15.010844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:15.010866 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:15.010992 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:15.010998 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.012502 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.012506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:15.013851 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:15.013863 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.015035 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.015038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:15.015141 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:15.015172 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 01:47:15.015294 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 01:47:15.015394 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":1}
I20260810 01:47:15.015623 20858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:15.009706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43208 (local address 127.9.77.131:39543)
0810 01:47:15.010164 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:15.010182 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:15.010215 (+ 33us) server_negotiation.cc:407] Connection header received
0810 01:47:15.010330 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:15.010337 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:15.010651 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:15.010891 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:15.011098 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.012366 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.013999 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.014889 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.015006 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:15.015186 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:15.015329 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 01:47:15.015451 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":176,"threads_started":1}
I20260810 01:47:15.016170 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:15.009186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:45485 (local address 127.0.0.1:55360)
0810 01:47:15.009537 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:15.009556 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:15.009662 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:15.011574 (+ 1912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:15.011578 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:15.011600 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:15.011703 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:15.011708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.013456 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.013460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:15.014497 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:15.014506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.015637 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.015642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:15.015794 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:15.015836 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 01:47:15.015896 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 01:47:15.015987 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":132,"threads_started":1}
I20260810 01:47:15.017105 20862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:15.010142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:46775 (local address 127.0.0.1:40912)
0810 01:47:15.011827 (+ 1685us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:15.011843 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:15.011938 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:15.012964 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:15.012969 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:15.012990 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:15.013089 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:15.013096 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.014270 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.014274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:15.015379 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:15.015390 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.016706 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.016710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:15.016797 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:15.016829 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 01:47:15.016902 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 01:47:15.016980 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1620,"thread_start_us":117,"threads_started":1}
I20260810 01:47:15.017426 20856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:15.009456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55360 (local address 127.9.77.129:45485)
0810 01:47:15.011024 (+ 1568us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:15.011034 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:15.011062 (+ 28us) server_negotiation.cc:407] Connection header received
0810 01:47:15.011158 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:15.011163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:15.011397 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:15.011568 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:15.011879 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.013311 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.014642 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.015484 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.016529 (+ 1045us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:15.016729 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:15.016803 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 01:47:15.016931 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1424,"thread_start_us":94,"threads_started":1}
I20260810 01:47:15.017979 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:15.010471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40912 (local address 127.9.77.130:46775)
0810 01:47:15.012250 (+ 1779us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:15.012262 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:15.012285 (+ 23us) server_negotiation.cc:407] Connection header received
0810 01:47:15.012368 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:15.012374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:15.012595 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:15.012758 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:15.013245 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.014122 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.015520 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.016399 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.016499 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:15.016641 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:15.017399 (+ 758us) server_negotiation.cc:299] Negotiation successful
0810 01:47:15.017499 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1656,"thread_start_us":124,"threads_started":1}
I20260810 01:47:15.019814 20702 tablet_service.cc:1467] Processing CreateTablet for tablet eab36c8a04694294b119a4003fa1dd7f (DEFAULT_TABLE table=test-table [id=572c287ee4da411c808f9e982c54d75c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:47:15.020603 20276 tablet_service.cc:1467] Processing CreateTablet for tablet eab36c8a04694294b119a4003fa1dd7f (DEFAULT_TABLE table=test-table [id=572c287ee4da411c808f9e982c54d75c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:47:15.020828 20493 tablet_service.cc:1467] Processing CreateTablet for tablet eab36c8a04694294b119a4003fa1dd7f (DEFAULT_TABLE table=test-table [id=572c287ee4da411c808f9e982c54d75c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:47:15.021242 20702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eab36c8a04694294b119a4003fa1dd7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:15.021879 20493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eab36c8a04694294b119a4003fa1dd7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:15.021881 20276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eab36c8a04694294b119a4003fa1dd7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:15.030198 20864 tablet_bootstrap.cc:492] T eab36c8a04694294b119a4003fa1dd7f P 49e30d0b090648f29deeb808001b14a7: Bootstrap starting.
I20260810 01:47:15.032011 20867 tablet_bootstrap.cc:492] T eab36c8a04694294b119a4003fa1dd7f P 12e99ab08944444b8ea106fa5aa38769: Bootstrap starting.
I20260810 01:47:15.032280 20866 tablet_bootstrap.cc:492] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19: Bootstrap starting.
I20260810 01:47:15.033557 20864 tablet_bootstrap.cc:654] T eab36c8a04694294b119a4003fa1dd7f P 49e30d0b090648f29deeb808001b14a7: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:15.034384 20866 tablet_bootstrap.cc:654] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:15.034435 20867 tablet_bootstrap.cc:654] T eab36c8a04694294b119a4003fa1dd7f P 12e99ab08944444b8ea106fa5aa38769: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:15.035207 20864 log.cc:824] T eab36c8a04694294b119a4003fa1dd7f P 49e30d0b090648f29deeb808001b14a7: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:15.035281 20867 log.cc:824] T eab36c8a04694294b119a4003fa1dd7f P 12e99ab08944444b8ea106fa5aa38769: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:15.035274 20866 log.cc:824] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:15.037642 20867 tablet_bootstrap.cc:492] T eab36c8a04694294b119a4003fa1dd7f P 12e99ab08944444b8ea106fa5aa38769: No bootstrap required, opened a new log
I20260810 01:47:15.037650 20866 tablet_bootstrap.cc:492] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19: No bootstrap required, opened a new log
I20260810 01:47:15.037650 20864 tablet_bootstrap.cc:492] T eab36c8a04694294b119a4003fa1dd7f P 49e30d0b090648f29deeb808001b14a7: No bootstrap required, opened a new log
I20260810 01:47:15.037914 20867 ts_tablet_manager.cc:1397] T eab36c8a04694294b119a4003fa1dd7f P 12e99ab08944444b8ea106fa5aa38769: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 01:47:15.037914 20866 ts_tablet_manager.cc:1397] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260810 01:47:15.037914 20864 ts_tablet_manager.cc:1397] T eab36c8a04694294b119a4003fa1dd7f P 49e30d0b090648f29deeb808001b14a7: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260810 01:47:15.042685 20866 raft_consensus.cc:348] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e30d0b090648f29deeb808001b14a7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39543 } } peers { permanent_uuid: "3ad0891514f242b8a78735404fbbcc19" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45485 } } peers { permanent_uuid: "12e99ab08944444b8ea106fa5aa38769" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 46775 } }
I20260810 01:47:15.043097 20866 raft_consensus.cc:374] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:15.043161 20866 raft_consensus.cc:729] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ad0891514f242b8a78735404fbbcc19, State: Initialized, Role: FOLLOWER
I20260810 01:47:15.042987 20864 raft_consensus.cc:348] T eab36c8a04694294b119a4003fa1dd7f P 49e30d0b090648f29deeb808001b14a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e30d0b090648f29deeb808001b14a7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39543 } } peers { permanent_uuid: "3ad0891514f242b8a78735404fbbcc19" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45485 } } peers { permanent_uuid: "12e99ab08944444b8ea106fa5aa38769" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 46775 } }
I20260810 01:47:15.043239 20864 raft_consensus.cc:374] T eab36c8a04694294b119a4003fa1dd7f P 49e30d0b090648f29deeb808001b14a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:15.043290 20864 raft_consensus.cc:729] T eab36c8a04694294b119a4003fa1dd7f P 49e30d0b090648f29deeb808001b14a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e30d0b090648f29deeb808001b14a7, State: Initialized, Role: FOLLOWER
I20260810 01:47:15.043149 20867 raft_consensus.cc:348] T eab36c8a04694294b119a4003fa1dd7f P 12e99ab08944444b8ea106fa5aa38769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e30d0b090648f29deeb808001b14a7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39543 } } peers { permanent_uuid: "3ad0891514f242b8a78735404fbbcc19" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45485 } } peers { permanent_uuid: "12e99ab08944444b8ea106fa5aa38769" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 46775 } }
I20260810 01:47:15.043507 20867 raft_consensus.cc:374] T eab36c8a04694294b119a4003fa1dd7f P 12e99ab08944444b8ea106fa5aa38769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:15.043577 20867 raft_consensus.cc:729] T eab36c8a04694294b119a4003fa1dd7f P 12e99ab08944444b8ea106fa5aa38769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e99ab08944444b8ea106fa5aa38769, State: Initialized, Role: FOLLOWER
I20260810 01:47:15.043766 20864 consensus_queue.cc:260] T eab36c8a04694294b119a4003fa1dd7f P 49e30d0b090648f29deeb808001b14a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e30d0b090648f29deeb808001b14a7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39543 } } peers { permanent_uuid: "3ad0891514f242b8a78735404fbbcc19" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45485 } } peers { permanent_uuid: "12e99ab08944444b8ea106fa5aa38769" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 46775 } }
I20260810 01:47:15.043767 20866 consensus_queue.cc:260] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e30d0b090648f29deeb808001b14a7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39543 } } peers { permanent_uuid: "3ad0891514f242b8a78735404fbbcc19" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45485 } } peers { permanent_uuid: "12e99ab08944444b8ea106fa5aa38769" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 46775 } }
I20260810 01:47:15.044138 20867 consensus_queue.cc:260] T eab36c8a04694294b119a4003fa1dd7f P 12e99ab08944444b8ea106fa5aa38769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e30d0b090648f29deeb808001b14a7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39543 } } peers { permanent_uuid: "3ad0891514f242b8a78735404fbbcc19" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45485 } } peers { permanent_uuid: "12e99ab08944444b8ea106fa5aa38769" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 46775 } }
I20260810 01:47:15.045106 20866 ts_tablet_manager.cc:1428] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 01:47:15.045106 20864 ts_tablet_manager.cc:1428] T eab36c8a04694294b119a4003fa1dd7f P 49e30d0b090648f29deeb808001b14a7: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 01:47:15.045343 20867 ts_tablet_manager.cc:1428] T eab36c8a04694294b119a4003fa1dd7f P 12e99ab08944444b8ea106fa5aa38769: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 01:47:15.045382 20809 heartbeater.cc:503] Master 127.9.77.190:33329 was elected leader, sending a full tablet report...
I20260810 01:47:15.046980 20597 heartbeater.cc:503] Master 127.9.77.190:33329 was elected leader, sending a full tablet report...
I20260810 01:47:15.048884 20377 heartbeater.cc:503] Master 127.9.77.190:33329 was elected leader, sending a full tablet report...
W20260810 01:47:15.163235 20382 tablet.cc:2369] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 01:47:15.181056 20600 tablet.cc:2369] T eab36c8a04694294b119a4003fa1dd7f P 12e99ab08944444b8ea106fa5aa38769: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 01:47:15.205724 20814 tablet.cc:2369] T eab36c8a04694294b119a4003fa1dd7f P 49e30d0b090648f29deeb808001b14a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 01:47:15.223845 20872 raft_consensus.cc:482] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:47:15.224225 20872 raft_consensus.cc:504] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e30d0b090648f29deeb808001b14a7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39543 } } peers { permanent_uuid: "3ad0891514f242b8a78735404fbbcc19" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45485 } } peers { permanent_uuid: "12e99ab08944444b8ea106fa5aa38769" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 46775 } }
I20260810 01:47:15.225986 20872 leader_election.cc:288] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49e30d0b090648f29deeb808001b14a7 (127.9.77.131:39543), 12e99ab08944444b8ea106fa5aa38769 (127.9.77.130:46775)
I20260810 01:47:15.231932 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:15.227050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:52421 (local address 127.9.77.130:46775)
0810 01:47:15.227209 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:15.227214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:15.227230 (+ 16us) server_negotiation.cc:407] Connection header received
0810 01:47:15.227292 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:15.227295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:15.227368 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:15.227464 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:15.228132 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.228776 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.230605 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.231249 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.231398 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:15.231465 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:15.231688 (+ 223us) server_negotiation.cc:299] Negotiation successful
0810 01:47:15.231780 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 01:47:15.231987 20877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:15.226304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:46775 (local address 127.9.77.129:52421)
0810 01:47:15.226851 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:15.226865 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:15.227007 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:15.227486 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:15.227489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:15.227501 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:15.227827 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:15.227834 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.228909 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.228914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:15.230468 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:15.230481 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.231406 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.231409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:15.231516 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:15.231542 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 01:47:15.231667 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 01:47:15.231764 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":164,"spinlock_wait_cycles":17024,"thread_start_us":146,"threads_started":1}
I20260810 01:47:15.233068 20878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:15.226263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:39543 (local address 127.9.77.129:57785)
0810 01:47:15.226758 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:15.226777 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:15.227008 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:15.227465 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:15.227469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:15.227488 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:15.227826 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:15.227834 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.229039 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.229042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:15.230468 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:15.230481 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.232613 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.232617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:15.232740 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:15.232763 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 01:47:15.232831 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 01:47:15.232909 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":161,"spinlock_wait_cycles":11520,"thread_start_us":175,"threads_started":1}
I20260810 01:47:15.233116 20858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:15.226383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:57785 (local address 127.9.77.131:39543)
0810 01:47:15.226572 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:15.226577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:15.226888 (+ 311us) server_negotiation.cc:407] Connection header received
0810 01:47:15.227171 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:15.227174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:15.227234 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:15.227278 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:15.227976 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.228916 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.231534 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.232230 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.232360 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:15.232412 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:15.232874 (+ 462us) server_negotiation.cc:299] Negotiation successful
0810 01:47:15.232980 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260810 01:47:15.233201 20522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab36c8a04694294b119a4003fa1dd7f" candidate_uuid: "3ad0891514f242b8a78735404fbbcc19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e99ab08944444b8ea106fa5aa38769" is_pre_election: true
I20260810 01:47:15.233675 20522 raft_consensus.cc:2454] T eab36c8a04694294b119a4003fa1dd7f P 12e99ab08944444b8ea106fa5aa38769 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ad0891514f242b8a78735404fbbcc19 in term 0.
I20260810 01:47:15.234314 20207 leader_election.cc:302] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e99ab08944444b8ea106fa5aa38769, 3ad0891514f242b8a78735404fbbcc19; no voters:
I20260810 01:47:15.234663 20872 raft_consensus.cc:2781] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:47:15.234676 20732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab36c8a04694294b119a4003fa1dd7f" candidate_uuid: "3ad0891514f242b8a78735404fbbcc19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e30d0b090648f29deeb808001b14a7" is_pre_election: true
I20260810 01:47:15.234781 20872 raft_consensus.cc:482] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:47:15.234831 20872 raft_consensus.cc:3037] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:15.235019 20732 raft_consensus.cc:2454] T eab36c8a04694294b119a4003fa1dd7f P 49e30d0b090648f29deeb808001b14a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ad0891514f242b8a78735404fbbcc19 in term 0.
I20260810 01:47:15.236382 20872 raft_consensus.cc:504] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e30d0b090648f29deeb808001b14a7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39543 } } peers { permanent_uuid: "3ad0891514f242b8a78735404fbbcc19" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45485 } } peers { permanent_uuid: "12e99ab08944444b8ea106fa5aa38769" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 46775 } }
I20260810 01:47:15.236824 20872 leader_election.cc:288] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19 [CANDIDATE]: Term 1 election: Requested vote from peers 49e30d0b090648f29deeb808001b14a7 (127.9.77.131:39543), 12e99ab08944444b8ea106fa5aa38769 (127.9.77.130:46775)
I20260810 01:47:15.237138 20732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab36c8a04694294b119a4003fa1dd7f" candidate_uuid: "3ad0891514f242b8a78735404fbbcc19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e30d0b090648f29deeb808001b14a7"
I20260810 01:47:15.237210 20522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab36c8a04694294b119a4003fa1dd7f" candidate_uuid: "3ad0891514f242b8a78735404fbbcc19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e99ab08944444b8ea106fa5aa38769"
I20260810 01:47:15.237274 20732 raft_consensus.cc:3037] T eab36c8a04694294b119a4003fa1dd7f P 49e30d0b090648f29deeb808001b14a7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:15.237362 20522 raft_consensus.cc:3037] T eab36c8a04694294b119a4003fa1dd7f P 12e99ab08944444b8ea106fa5aa38769 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:15.238798 20522 raft_consensus.cc:2454] T eab36c8a04694294b119a4003fa1dd7f P 12e99ab08944444b8ea106fa5aa38769 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ad0891514f242b8a78735404fbbcc19 in term 1.
I20260810 01:47:15.238798 20732 raft_consensus.cc:2454] T eab36c8a04694294b119a4003fa1dd7f P 49e30d0b090648f29deeb808001b14a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ad0891514f242b8a78735404fbbcc19 in term 1.
I20260810 01:47:15.239146 20204 leader_election.cc:302] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ad0891514f242b8a78735404fbbcc19, 49e30d0b090648f29deeb808001b14a7; no voters:
I20260810 01:47:15.239342 20872 raft_consensus.cc:2781] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:47:15.239640 20872 raft_consensus.cc:686] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19 [term 1 LEADER]: Becoming Leader. State: Replica: 3ad0891514f242b8a78735404fbbcc19, State: Running, Role: LEADER
I20260810 01:47:15.240130 20872 consensus_queue.cc:237] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e30d0b090648f29deeb808001b14a7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39543 } } peers { permanent_uuid: "3ad0891514f242b8a78735404fbbcc19" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45485 } } peers { permanent_uuid: "12e99ab08944444b8ea106fa5aa38769" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 46775 } }
I20260810 01:47:15.245045 18889 catalog_manager.cc:5495] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ad0891514f242b8a78735404fbbcc19 (127.9.77.129). New cstate: current_term: 1 leader_uuid: "3ad0891514f242b8a78735404fbbcc19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e30d0b090648f29deeb808001b14a7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ad0891514f242b8a78735404fbbcc19" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12e99ab08944444b8ea106fa5aa38769" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 46775 } health_report { overall_health: UNKNOWN } } }
I20260810 01:47:15.318764 20856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:15.314582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55364 (local address 127.9.77.129:45485)
0810 01:47:15.314795 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:15.314799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:15.314812 (+ 13us) server_negotiation.cc:407] Connection header received
0810 01:47:15.314915 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:15.314917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:15.314989 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:15.315123 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:47:15.315427 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.316002 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.317021 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.317254 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.317371 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:15.317720 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:47:15.318306 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:47:15.318461 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:15.318494 (+ 33us) server_negotiation.cc:299] Negotiation successful
0810 01:47:15.318565 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 01:47:15.326994 20732 raft_consensus.cc:1261] T eab36c8a04694294b119a4003fa1dd7f P 49e30d0b090648f29deeb808001b14a7 [term 1 FOLLOWER]: Refusing update from remote peer 3ad0891514f242b8a78735404fbbcc19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:47:15.327178 20522 raft_consensus.cc:1261] T eab36c8a04694294b119a4003fa1dd7f P 12e99ab08944444b8ea106fa5aa38769 [term 1 FOLLOWER]: Refusing update from remote peer 3ad0891514f242b8a78735404fbbcc19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:47:15.328119 20879 consensus_queue.cc:1035] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e30d0b090648f29deeb808001b14a7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:47:15.328487 20872 consensus_queue.cc:1035] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e99ab08944444b8ea106fa5aa38769" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 46775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:47:15.340204 20887 mvcc.cc:205] Tried to move back new op lower bound from 7316793079089885184 to 7316793078746992640. Current Snapshot: MvccSnapshot[applied={T|T < 7316793079089885184}]
I20260810 01:47:15.341787 20885 mvcc.cc:205] Tried to move back new op lower bound from 7316793079089885184 to 7316793078746992640. Current Snapshot: MvccSnapshot[applied={T|T < 7316793079089885184}]
I20260810 01:47:15.354853 18889 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:47:15.355132 18889 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:47:15.361876 18889 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ae815b0781f4d19aba6d526311d18c9: Sending DeleteTablet for 3 replicas of tablet eab36c8a04694294b119a4003fa1dd7f
I20260810 01:47:15.362891 20276 tablet_service.cc:1514] Processing DeleteTablet for tablet eab36c8a04694294b119a4003fa1dd7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:47:15 UTC) from {username='oryx', principal='oryx/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:55360
I20260810 01:47:15.362840 20702 tablet_service.cc:1514] Processing DeleteTablet for tablet eab36c8a04694294b119a4003fa1dd7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:47:15 UTC) from {username='oryx', principal='oryx/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:43208
I20260810 01:47:15.363570 20901 tablet_replica.cc:330] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19: stopping tablet replica
I20260810 01:47:15.363907 20493 tablet_service.cc:1514] Processing DeleteTablet for tablet eab36c8a04694294b119a4003fa1dd7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:47:15 UTC) from {username='oryx', principal='oryx/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:40912
I20260810 01:47:15.364229 20901 raft_consensus.cc:2229] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19 [term 1 LEADER]: Raft consensus shutting down.
I20260810 01:47:15.364821 20901 raft_consensus.cc:2258] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:47:15.365332 20903 tablet_replica.cc:330] T eab36c8a04694294b119a4003fa1dd7f P 49e30d0b090648f29deeb808001b14a7: stopping tablet replica
I20260810 01:47:15.365713 20903 raft_consensus.cc:2229] T eab36c8a04694294b119a4003fa1dd7f P 49e30d0b090648f29deeb808001b14a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:47:15.366118 20903 raft_consensus.cc:2258] T eab36c8a04694294b119a4003fa1dd7f P 49e30d0b090648f29deeb808001b14a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:47:15.367165 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 18838
I20260810 01:47:15.368117 20903 ts_tablet_manager.cc:1905] T eab36c8a04694294b119a4003fa1dd7f P 49e30d0b090648f29deeb808001b14a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:47:15.377321 20903 ts_tablet_manager.cc:1918] T eab36c8a04694294b119a4003fa1dd7f P 49e30d0b090648f29deeb808001b14a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:47:15.377490 20903 log.cc:1196] T eab36c8a04694294b119a4003fa1dd7f P 49e30d0b090648f29deeb808001b14a7: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/wal/wals/eab36c8a04694294b119a4003fa1dd7f
I20260810 01:47:15.377933 20903 ts_tablet_manager.cc:1939] T eab36c8a04694294b119a4003fa1dd7f P 49e30d0b090648f29deeb808001b14a7: Deleting consensus metadata
I20260810 01:47:15.385102 20901 ts_tablet_manager.cc:1905] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:47:15.388401 20906 tablet_replica.cc:330] T eab36c8a04694294b119a4003fa1dd7f P 12e99ab08944444b8ea106fa5aa38769: stopping tablet replica
I20260810 01:47:15.389039 20906 raft_consensus.cc:2229] T eab36c8a04694294b119a4003fa1dd7f P 12e99ab08944444b8ea106fa5aa38769 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:47:15.389678 20906 raft_consensus.cc:2258] T eab36c8a04694294b119a4003fa1dd7f P 12e99ab08944444b8ea106fa5aa38769 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:47:15.390179 20901 ts_tablet_manager.cc:1918] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:47:15.390327 20901 log.cc:1196] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-0/wal/wals/eab36c8a04694294b119a4003fa1dd7f
I20260810 01:47:15.390769 20901 ts_tablet_manager.cc:1939] T eab36c8a04694294b119a4003fa1dd7f P 3ad0891514f242b8a78735404fbbcc19: Deleting consensus metadata
I20260810 01:47:15.392613 20906 ts_tablet_manager.cc:1905] T eab36c8a04694294b119a4003fa1dd7f P 12e99ab08944444b8ea106fa5aa38769: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:47:15.392787 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:15.366015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45832 (local address 127.9.77.189:38111)
0810 01:47:15.366141 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:15.366146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:15.367246 (+ 1100us) server_negotiation.cc:407] Connection header received
0810 01:47:15.380167 (+ 12921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:15.380172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:15.380264 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:15.380328 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:15.381771 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.382683 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.384248 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.384630 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.384719 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:15.387907 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:15.387911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:15.387915 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:15.387953 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:15.390076 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:15.390707 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:15.390714 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:15.390719 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:15.390841 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:15.391207 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:15.391211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:15.391213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:15.391517 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:15.391636 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:15.392238 (+ 602us) server_negotiation.cc:299] Negotiation successful
0810 01:47:15.392578 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260810 01:47:15.394232 20201 connection.cc:535] server connection from 127.0.0.1:55360 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 01:47:15.395010 20629 connection.cc:535] server connection from 127.0.0.1:43208 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 01:47:15.396579 20906 ts_tablet_manager.cc:1918] T eab36c8a04694294b119a4003fa1dd7f P 12e99ab08944444b8ea106fa5aa38769: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:47:15.396698 20906 log.cc:1196] T eab36c8a04694294b119a4003fa1dd7f P 12e99ab08944444b8ea106fa5aa38769: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/wal/wals/eab36c8a04694294b119a4003fa1dd7f
I20260810 01:47:15.397094 20906 ts_tablet_manager.cc:1939] T eab36c8a04694294b119a4003fa1dd7f P 12e99ab08944444b8ea106fa5aa38769: Deleting consensus metadata
W20260810 01:47:15.398604 20421 connection.cc:441] server connection from 127.0.0.1:40912 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40912 (request call id 1) could send its response
I20260810 01:47:15.405637 20385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:15.366007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51084 (local address 127.9.77.188:45735)
0810 01:47:15.366148 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:15.366154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:15.394896 (+ 28742us) server_negotiation.cc:407] Connection header received
0810 01:47:15.395156 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:15.395160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:15.395254 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:15.395324 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:15.396452 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.397371 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.398513 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.398859 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.398960 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:15.400999 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:15.401003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:15.401006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:15.401037 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:15.403107 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:15.403667 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:15.403671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:15.403672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:15.403768 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:15.404105 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:15.404108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:15.404110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:15.404355 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:15.404479 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:15.405317 (+ 838us) server_negotiation.cc:299] Negotiation successful
0810 01:47:15.405454 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 01:47:15.410813 20825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:15.400143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51098 (local address 127.9.77.188:45735)
0810 01:47:15.400280 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:15.400284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:15.400554 (+ 270us) server_negotiation.cc:407] Connection header received
0810 01:47:15.400722 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:15.400726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:15.400777 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:15.400829 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:15.401745 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.402554 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.403792 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.404172 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.404520 (+ 348us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:15.406958 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:15.406962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:15.406965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:15.406990 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:15.408878 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:15.409395 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:15.409409 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:15.409411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:15.409492 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:15.409785 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:15.409789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:15.409791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:15.409984 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:15.410108 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:15.410512 (+ 404us) server_negotiation.cc:299] Negotiation successful
0810 01:47:15.410632 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":251,"server-negotiator.queue_time_us":59}
I20260810 01:47:15.411633 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:15.401132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45848 (local address 127.9.77.189:38111)
0810 01:47:15.401296 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:15.401303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:15.402318 (+ 1015us) server_negotiation.cc:407] Connection header received
0810 01:47:15.402616 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:15.402621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:15.402680 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:15.402736 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:15.403674 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.404478 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.405644 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:15.405972 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:15.406065 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:15.407732 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:15.407736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:15.407747 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:15.407777 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:15.409553 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:15.410056 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:15.410059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:15.410061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:15.410144 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:15.410422 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:15.410425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:15.410427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:15.410622 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:15.410712 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:15.411353 (+ 641us) server_negotiation.cc:299] Negotiation successful
0810 01:47:15.411481 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 01:47:16.344271 20919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:16.343354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.9.77.131:60057)
0810 01:47:16.343870 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:16.344007 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":115,"threads_started":1}
I20260810 01:47:16.344388 20922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:16.343729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.9.77.129:54533)
0810 01:47:16.344142 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:16.344244 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":86,"threads_started":1}
W20260810 01:47:16.344749 20377 heartbeater.cc:644] Failed to heartbeat to 127.9.77.190:33329 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
W20260810 01:47:16.345011 20809 heartbeater.cc:644] Failed to heartbeat to 127.9.77.190:33329 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
I20260810 01:47:16.346570 20923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:16.345976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.9.77.130:43531)
0810 01:47:16.346342 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:16.346467 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":112,"threads_started":1}
W20260810 01:47:16.346874 20597 heartbeater.cc:644] Failed to heartbeat to 127.9.77.190:33329 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
I20260810 01:47:16.864779 20928 raft_consensus.cc:482] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9ae815b0781f4d19aba6d526311d18c9)
I20260810 01:47:16.864779 20927 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9ae815b0781f4d19aba6d526311d18c9)
I20260810 01:47:16.865144 20928 raft_consensus.cc:504] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: false } }
I20260810 01:47:16.865144 20927 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: false } }
I20260810 01:47:16.866744 20927 leader_election.cc:288] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ae815b0781f4d19aba6d526311d18c9 (127.9.77.190:33329), 646e25980e154c5c8b8c24f9040a9bb5 (127.9.77.188:45735)
I20260810 01:47:16.866745 20928 leader_election.cc:288] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ae815b0781f4d19aba6d526311d18c9 (127.9.77.190:33329), b3e994dc2a3d412ea8f13f43c39bc783 (127.9.77.189:38111)
I20260810 01:47:16.867810 20932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:16.867131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.0.0.1:37042)
0810 01:47:16.867561 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:16.867667 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":95,"threads_started":1}
I20260810 01:47:16.868219 20934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:16.867010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.0.0.1:37030)
0810 01:47:16.867993 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:16.868103 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":911,"mutex_wait_us":279,"thread_start_us":77,"threads_started":1}
W20260810 01:47:16.868177 19781 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
W20260810 01:47:16.868376 20076 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
I20260810 01:47:16.869354 20934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:16.869105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.0.0.1:37050)
0810 01:47:16.869237 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:16.869284 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 01:47:16.869364 20932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:16.869105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.0.0.1:37048)
0810 01:47:16.869237 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:16.869283 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260810 01:47:16.869509 20076 leader_election.cc:334] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9ae815b0781f4d19aba6d526311d18c9 (127.9.77.190:33329): Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
W20260810 01:47:16.869573 19781 leader_election.cc:334] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9ae815b0781f4d19aba6d526311d18c9 (127.9.77.190:33329): Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
I20260810 01:47:16.872288 20933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:16.867187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45860 (local address 127.9.77.189:38111)
0810 01:47:16.867784 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:16.867790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:16.867808 (+ 18us) server_negotiation.cc:407] Connection header received
0810 01:47:16.867901 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:16.867904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:16.867959 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:16.868437 (+ 478us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:16.868511 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:16.869408 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:16.871051 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:16.871620 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:16.871799 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:16.871881 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:16.872088 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 01:47:16.872164 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":532,"thread_start_us":60,"threads_started":1}
I20260810 01:47:16.872303 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:16.867131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:45735 (local address 127.0.0.1:51108)
0810 01:47:16.867653 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:16.867667 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:16.867796 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:16.868359 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:16.868363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:16.868386 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:16.868508 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:16.868514 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:16.869545 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:16.869548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:16.870920 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:16.870931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:16.871725 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:16.871730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:16.871914 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:16.871933 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 01:47:16.872031 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 01:47:16.872146 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":251,"thread_start_us":62,"threads_started":1}
I20260810 01:47:16.872303 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:16.867568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51108 (local address 127.9.77.188:45735)
0810 01:47:16.868012 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:16.868065 (+ 53us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:16.868078 (+ 13us) server_negotiation.cc:407] Connection header received
0810 01:47:16.868123 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:16.868126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:16.868180 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:16.868258 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:16.868628 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:16.869420 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:16.871096 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:16.871595 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:16.871722 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:16.871816 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:16.872082 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 01:47:16.872169 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"spinlock_wait_cycles":10880,"thread_start_us":61,"threads_started":1}
I20260810 01:47:16.872282 20930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:16.867009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:38111 (local address 127.0.0.1:45860)
0810 01:47:16.867635 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:16.867654 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:16.867799 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:16.868124 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:16.868127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:16.868156 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:16.868337 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:16.868344 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:16.869547 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:16.869550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:16.870920 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:16.870932 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:16.871760 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:16.871764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:16.871866 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:16.871891 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 01:47:16.872001 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 01:47:16.872122 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"spinlock_wait_cycles":384,"thread_start_us":106,"threads_started":1}
I20260810 01:47:16.873044 20108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "646e25980e154c5c8b8c24f9040a9bb5" is_pre_election: true
I20260810 01:47:16.873145 19820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "646e25980e154c5c8b8c24f9040a9bb5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" is_pre_election: true
I20260810 01:47:16.873358 20108 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3e994dc2a3d412ea8f13f43c39bc783 in term 1.
I20260810 01:47:16.873385 19820 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 646e25980e154c5c8b8c24f9040a9bb5 in term 1.
I20260810 01:47:16.873780 19786 leader_election.cc:302] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 646e25980e154c5c8b8c24f9040a9bb5, b3e994dc2a3d412ea8f13f43c39bc783; no voters: 9ae815b0781f4d19aba6d526311d18c9
I20260810 01:47:16.873817 20078 leader_election.cc:302] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 646e25980e154c5c8b8c24f9040a9bb5, b3e994dc2a3d412ea8f13f43c39bc783; no voters: 9ae815b0781f4d19aba6d526311d18c9
I20260810 01:47:16.874069 20928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 01:47:16.874080 20927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 01:47:16.874130 20928 raft_consensus.cc:482] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9ae815b0781f4d19aba6d526311d18c9)
I20260810 01:47:16.874137 20927 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9ae815b0781f4d19aba6d526311d18c9)
I20260810 01:47:16.874171 20928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [term 1 FOLLOWER]: Advancing to term 2
I20260810 01:47:16.874176 20927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 1 FOLLOWER]: Advancing to term 2
I20260810 01:47:16.875463 20928 raft_consensus.cc:504] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: false } }
I20260810 01:47:16.875463 20927 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: false } }
I20260810 01:47:16.875869 20927 leader_election.cc:288] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [CANDIDATE]: Term 2 election: Requested vote from peers 9ae815b0781f4d19aba6d526311d18c9 (127.9.77.190:33329), 646e25980e154c5c8b8c24f9040a9bb5 (127.9.77.188:45735)
I20260810 01:47:16.875893 20928 leader_election.cc:288] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [CANDIDATE]: Term 2 election: Requested vote from peers 9ae815b0781f4d19aba6d526311d18c9 (127.9.77.190:33329), b3e994dc2a3d412ea8f13f43c39bc783 (127.9.77.189:38111)
I20260810 01:47:16.876454 20108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "646e25980e154c5c8b8c24f9040a9bb5"
I20260810 01:47:16.876566 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:16.876268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.0.0.1:37076)
0810 01:47:16.876413 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:16.876483 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 01:47:16.876648 19820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "646e25980e154c5c8b8c24f9040a9bb5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b3e994dc2a3d412ea8f13f43c39bc783"
I20260810 01:47:16.876608 20930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:16.876038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.0.0.1:37074)
0810 01:47:16.876475 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:16.876530 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357}
I20260810 01:47:16.876647 20108 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b3e994dc2a3d412ea8f13f43c39bc783 in current term 2: Already voted for candidate 646e25980e154c5c8b8c24f9040a9bb5 in this term.
I20260810 01:47:16.876813 19820 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 646e25980e154c5c8b8c24f9040a9bb5 in current term 2: Already voted for candidate b3e994dc2a3d412ea8f13f43c39bc783 in this term.
I20260810 01:47:16.877290 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:16.877070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.0.0.1:37092)
0810 01:47:16.877163 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:16.877215 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260810 01:47:16.877396 19781 leader_election.cc:334] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9ae815b0781f4d19aba6d526311d18c9 (127.9.77.190:33329): Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
I20260810 01:47:16.877480 19781 leader_election.cc:302] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3e994dc2a3d412ea8f13f43c39bc783; no voters: 646e25980e154c5c8b8c24f9040a9bb5, 9ae815b0781f4d19aba6d526311d18c9
I20260810 01:47:16.877624 20927 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 01:47:16.877794 20930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:16.877562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.0.0.1:37106)
0810 01:47:16.877665 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:16.877723 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 01:47:16.877912 20076 leader_election.cc:334] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9ae815b0781f4d19aba6d526311d18c9 (127.9.77.190:33329): Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
I20260810 01:47:16.878028 20076 leader_election.cc:302] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 646e25980e154c5c8b8c24f9040a9bb5; no voters: 9ae815b0781f4d19aba6d526311d18c9, b3e994dc2a3d412ea8f13f43c39bc783
I20260810 01:47:16.878193 20928 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 01:47:17.346324 20941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:17.345643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.9.77.129:49971)
0810 01:47:17.346058 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:17.346150 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":145,"threads_started":1}
I20260810 01:47:17.347347 20942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:17.345731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.9.77.131:34289)
0810 01:47:17.347145 (+ 1414us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:17.347228 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1323,"thread_start_us":100,"threads_started":1}
I20260810 01:47:17.348191 20944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:17.347617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.9.77.130:33293)
0810 01:47:17.347995 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:17.348093 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":103,"threads_started":1}
I20260810 01:47:18.348620 20945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.347435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.9.77.129:38569)
0810 01:47:18.348362 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:18.348464 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":812,"thread_start_us":150,"threads_started":1}
I20260810 01:47:18.349733 20948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.349204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.9.77.131:49639)
0810 01:47:18.349542 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:18.349612 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":98,"threads_started":1}
I20260810 01:47:18.350135 20946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.349137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.9.77.130:58419)
0810 01:47:18.349924 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:18.350001 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":87,"threads_started":1}
I20260810 01:47:18.396405 20949 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:47:18.396631 20949 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: false } }
I20260810 01:47:18.397293 20949 leader_election.cc:288] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9ae815b0781f4d19aba6d526311d18c9 (127.9.77.190:33329), 646e25980e154c5c8b8c24f9040a9bb5 (127.9.77.188:45735)
I20260810 01:47:18.397997 20108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "646e25980e154c5c8b8c24f9040a9bb5" is_pre_election: true
I20260810 01:47:18.398111 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.397588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.0.0.1:37304)
0810 01:47:18.397865 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:18.397954 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":99,"threads_started":1}
I20260810 01:47:18.398205 20108 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3e994dc2a3d412ea8f13f43c39bc783 in term 2.
I20260810 01:47:18.398651 19786 leader_election.cc:302] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 646e25980e154c5c8b8c24f9040a9bb5, b3e994dc2a3d412ea8f13f43c39bc783; no voters:
I20260810 01:47:18.398866 20949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 01:47:18.398947 20949 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:47:18.399006 20949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 2 FOLLOWER]: Advancing to term 3
I20260810 01:47:18.399092 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.398774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.0.0.1:37316)
0810 01:47:18.398926 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:18.399008 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 01:47:18.399215 19781 leader_election.cc:334] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9ae815b0781f4d19aba6d526311d18c9 (127.9.77.190:33329): Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
I20260810 01:47:18.400609 20949 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: false } }
I20260810 01:47:18.401028 20949 leader_election.cc:288] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [CANDIDATE]: Term 3 election: Requested vote from peers 9ae815b0781f4d19aba6d526311d18c9 (127.9.77.190:33329), 646e25980e154c5c8b8c24f9040a9bb5 (127.9.77.188:45735)
I20260810 01:47:18.401464 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.401171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.0.0.1:37322)
0810 01:47:18.401293 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:18.401353 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260810 01:47:18.401558 20108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "646e25980e154c5c8b8c24f9040a9bb5"
I20260810 01:47:18.401747 20108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [term 2 FOLLOWER]: Advancing to term 3
I20260810 01:47:18.402297 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.402065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.0.0.1:37336)
0810 01:47:18.402181 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:18.402242 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 01:47:18.402420 19781 leader_election.cc:334] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9ae815b0781f4d19aba6d526311d18c9 (127.9.77.190:33329): Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
I20260810 01:47:18.403267 20108 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e994dc2a3d412ea8f13f43c39bc783 in term 3.
I20260810 01:47:18.403623 19786 leader_election.cc:302] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 646e25980e154c5c8b8c24f9040a9bb5, b3e994dc2a3d412ea8f13f43c39bc783; no voters: 9ae815b0781f4d19aba6d526311d18c9
I20260810 01:47:18.403913 20949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 01:47:18.404194 20949 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [term 3 LEADER]: Becoming Leader. State: Replica: b3e994dc2a3d412ea8f13f43c39bc783, State: Running, Role: LEADER
I20260810 01:47:18.404551 20949 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: false } }
I20260810 01:47:18.405813 20953 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3e994dc2a3d412ea8f13f43c39bc783. Latest consensus state: current_term: 3 leader_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: false } } }
I20260810 01:47:18.405970 20953 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:18.406332 20958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:47:18.409649 20958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:47:18.410140 20958 catalog_manager.cc:1259] Loaded cluster ID: 30255652128446de977d487ec2e58d4b
I20260810 01:47:18.410198 20958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:47:18.410962 20958 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:47:18.411405 20958 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783: Loaded TSK: 0
I20260810 01:47:18.412447 20958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:47:18.421619 19807 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 01:47:18.425805 19807 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:47:18.429749 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.429432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.0.0.1:37346)
0810 01:47:18.429582 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:18.429655 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 01:47:18.429937 20108 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [term 3 FOLLOWER]: Refusing update from remote peer b3e994dc2a3d412ea8f13f43c39bc783: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260810 01:47:18.430636 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.430372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.0.0.1:37360)
0810 01:47:18.430467 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:18.430549 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 01:47:18.430583 20949 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260810 01:47:18.430788 19781 consensus_peers.cc:487] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 -> Peer 9ae815b0781f4d19aba6d526311d18c9 (127.9.77.190:33329): Couldn't send request to peer 9ae815b0781f4d19aba6d526311d18c9. Status: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 01:47:18.432459 20961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3e994dc2a3d412ea8f13f43c39bc783. Latest consensus state: current_term: 3 leader_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: false } } }
I20260810 01:47:18.432660 20961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 01:47:18.434300 20953 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: false } } }
I20260810 01:47:18.434468 20953 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783 [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:18.434689 20961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ae815b0781f4d19aba6d526311d18c9" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33329 } } peers { permanent_uuid: "b3e994dc2a3d412ea8f13f43c39bc783" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 38111 } attrs { promote: false } } peers { permanent_uuid: "646e25980e154c5c8b8c24f9040a9bb5" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 45735 } attrs { promote: false } } }
I20260810 01:47:18.434839 20961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 646e25980e154c5c8b8c24f9040a9bb5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 01:47:18.447355 20970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.442199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:46775 (local address 127.0.0.1:40928)
0810 01:47:18.442457 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:18.442469 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:18.442552 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:18.443170 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:18.443174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:18.443193 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:18.443515 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:18.443521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:18.444569 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:18.444575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:18.445888 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:18.445899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:18.446812 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:18.446816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:18.446934 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:18.446970 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 01:47:18.447040 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 01:47:18.447146 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":167,"thread_start_us":119,"threads_started":1}
I20260810 01:47:18.447570 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.441894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:39543 (local address 127.0.0.1:43220)
0810 01:47:18.442328 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:18.442348 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:18.442461 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:18.443223 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:18.443226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:18.443243 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:18.443395 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:18.443404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:18.444925 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:18.444929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:18.446120 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:18.446128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:18.446997 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:18.447003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:18.447094 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:18.447119 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 01:47:18.447197 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 01:47:18.447287 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":52}
I20260810 01:47:18.448004 20968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.442025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43220 (local address 127.9.77.131:39543)
0810 01:47:18.442913 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:18.442919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:18.442935 (+ 16us) server_negotiation.cc:407] Connection header received
0810 01:47:18.443002 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:18.443004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:18.443071 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:18.443669 (+ 598us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:18.443854 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:18.444695 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:18.446247 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:18.446835 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:18.446980 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:18.447075 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:18.447239 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 01:47:18.447334 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":102,"threads_started":1}
I20260810 01:47:18.447875 20975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.442535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40928 (local address 127.9.77.130:46775)
0810 01:47:18.442868 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:18.442873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:18.442891 (+ 18us) server_negotiation.cc:407] Connection header received
0810 01:47:18.442956 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:18.442959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:18.443023 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:18.443340 (+ 317us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:18.443635 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:18.444459 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:18.446020 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:18.446681 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:18.446795 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:18.446880 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:18.447606 (+ 726us) server_negotiation.cc:299] Negotiation successful
0810 01:47:18.447701 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":103,"threads_started":1}
I20260810 01:47:18.448091 20977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.442931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55378 (local address 127.9.77.129:45485)
0810 01:47:18.443267 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:18.443273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:18.443311 (+ 38us) server_negotiation.cc:407] Connection header received
0810 01:47:18.443435 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:18.443440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:18.443507 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:18.443606 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:18.443858 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:18.445023 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:18.446268 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:18.446922 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:18.447033 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:18.447110 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:18.447722 (+ 612us) server_negotiation.cc:299] Negotiation successful
0810 01:47:18.447862 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":97,"threads_started":1}
I20260810 01:47:18.448810 20493 tablet_service.cc:1467] Processing CreateTablet for tablet b34233e628544d15bd571d56c67bc734 (DEFAULT_TABLE table=test-table [id=8559a9669bd64a80b06e250b547c93c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:47:18.449002 20702 tablet_service.cc:1467] Processing CreateTablet for tablet b34233e628544d15bd571d56c67bc734 (DEFAULT_TABLE table=test-table [id=8559a9669bd64a80b06e250b547c93c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:47:18.449257 20493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b34233e628544d15bd571d56c67bc734. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:18.449317 20702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b34233e628544d15bd571d56c67bc734. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:18.452153 20978 tablet_bootstrap.cc:492] T b34233e628544d15bd571d56c67bc734 P 49e30d0b090648f29deeb808001b14a7: Bootstrap starting.
I20260810 01:47:18.452306 20979 tablet_bootstrap.cc:492] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769: Bootstrap starting.
I20260810 01:47:18.452759 20976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.442718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:45485 (local address 127.0.0.1:55378)
0810 01:47:18.443233 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:18.443245 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:18.443337 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:18.443625 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:18.443628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:18.443645 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:18.443732 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:18.443739 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:18.445134 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:18.445139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:18.446124 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:18.446134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:18.447169 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:18.447175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:18.447262 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:18.447280 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 01:47:18.447344 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 01:47:18.447412 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":1,"thread_start_us":103,"threads_started":1}
I20260810 01:47:18.453503 20978 tablet_bootstrap.cc:654] T b34233e628544d15bd571d56c67bc734 P 49e30d0b090648f29deeb808001b14a7: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:18.453500 20276 tablet_service.cc:1467] Processing CreateTablet for tablet b34233e628544d15bd571d56c67bc734 (DEFAULT_TABLE table=test-table [id=8559a9669bd64a80b06e250b547c93c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:47:18.453702 20979 tablet_bootstrap.cc:654] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:18.453888 20276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b34233e628544d15bd571d56c67bc734. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:18.455013 20978 tablet_bootstrap.cc:492] T b34233e628544d15bd571d56c67bc734 P 49e30d0b090648f29deeb808001b14a7: No bootstrap required, opened a new log
I20260810 01:47:18.455116 20978 ts_tablet_manager.cc:1397] T b34233e628544d15bd571d56c67bc734 P 49e30d0b090648f29deeb808001b14a7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 01:47:18.455212 20979 tablet_bootstrap.cc:492] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769: No bootstrap required, opened a new log
I20260810 01:47:18.455335 20979 ts_tablet_manager.cc:1397] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 01:47:18.455686 20978 raft_consensus.cc:348] T b34233e628544d15bd571d56c67bc734 P 49e30d0b090648f29deeb808001b14a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e30d0b090648f29deeb808001b14a7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39543 } } peers { permanent_uuid: "12e99ab08944444b8ea106fa5aa38769" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 46775 } } peers { permanent_uuid: "3ad0891514f242b8a78735404fbbcc19" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45485 } }
I20260810 01:47:18.455829 20978 raft_consensus.cc:374] T b34233e628544d15bd571d56c67bc734 P 49e30d0b090648f29deeb808001b14a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:18.455866 20978 raft_consensus.cc:729] T b34233e628544d15bd571d56c67bc734 P 49e30d0b090648f29deeb808001b14a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e30d0b090648f29deeb808001b14a7, State: Initialized, Role: FOLLOWER
I20260810 01:47:18.455957 20979 raft_consensus.cc:348] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e30d0b090648f29deeb808001b14a7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39543 } } peers { permanent_uuid: "12e99ab08944444b8ea106fa5aa38769" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 46775 } } peers { permanent_uuid: "3ad0891514f242b8a78735404fbbcc19" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45485 } }
I20260810 01:47:18.456058 20978 consensus_queue.cc:260] T b34233e628544d15bd571d56c67bc734 P 49e30d0b090648f29deeb808001b14a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e30d0b090648f29deeb808001b14a7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39543 } } peers { permanent_uuid: "12e99ab08944444b8ea106fa5aa38769" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 46775 } } peers { permanent_uuid: "3ad0891514f242b8a78735404fbbcc19" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45485 } }
I20260810 01:47:18.456131 20979 raft_consensus.cc:374] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:18.456174 20979 raft_consensus.cc:729] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e99ab08944444b8ea106fa5aa38769, State: Initialized, Role: FOLLOWER
I20260810 01:47:18.456331 20979 consensus_queue.cc:260] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e30d0b090648f29deeb808001b14a7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39543 } } peers { permanent_uuid: "12e99ab08944444b8ea106fa5aa38769" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 46775 } } peers { permanent_uuid: "3ad0891514f242b8a78735404fbbcc19" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45485 } }
I20260810 01:47:18.456501 20978 ts_tablet_manager.cc:1428] T b34233e628544d15bd571d56c67bc734 P 49e30d0b090648f29deeb808001b14a7: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 01:47:18.456753 20979 ts_tablet_manager.cc:1428] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 01:47:18.457352 20948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.457060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.9.77.131:56147)
0810 01:47:18.457188 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:18.457265 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 01:47:18.458069 20946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.457489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.9.77.130:37993)
0810 01:47:18.457909 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:18.457970 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332}
I20260810 01:47:18.460978 20948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.460680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.9.77.131:52545)
0810 01:47:18.460814 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:18.460887 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 01:47:18.461706 20811 heartbeater.cc:503] Master 127.9.77.189:38111 was elected leader, sending a full tablet report...
I20260810 01:47:18.462222 20988 tablet_bootstrap.cc:492] T b34233e628544d15bd571d56c67bc734 P 3ad0891514f242b8a78735404fbbcc19: Bootstrap starting.
I20260810 01:47:18.463730 20988 tablet_bootstrap.cc:654] T b34233e628544d15bd571d56c67bc734 P 3ad0891514f242b8a78735404fbbcc19: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:18.463879 20599 heartbeater.cc:503] Master 127.9.77.189:38111 was elected leader, sending a full tablet report...
I20260810 01:47:18.464397 20946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.463863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.9.77.130:52653)
0810 01:47:18.464226 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:18.464297 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277}
I20260810 01:47:18.465734 20988 tablet_bootstrap.cc:492] T b34233e628544d15bd571d56c67bc734 P 3ad0891514f242b8a78735404fbbcc19: No bootstrap required, opened a new log
I20260810 01:47:18.465873 20988 ts_tablet_manager.cc:1397] T b34233e628544d15bd571d56c67bc734 P 3ad0891514f242b8a78735404fbbcc19: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260810 01:47:18.466351 20988 raft_consensus.cc:348] T b34233e628544d15bd571d56c67bc734 P 3ad0891514f242b8a78735404fbbcc19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e30d0b090648f29deeb808001b14a7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39543 } } peers { permanent_uuid: "12e99ab08944444b8ea106fa5aa38769" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 46775 } } peers { permanent_uuid: "3ad0891514f242b8a78735404fbbcc19" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45485 } }
I20260810 01:47:18.466504 20988 raft_consensus.cc:374] T b34233e628544d15bd571d56c67bc734 P 3ad0891514f242b8a78735404fbbcc19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:18.466552 20988 raft_consensus.cc:729] T b34233e628544d15bd571d56c67bc734 P 3ad0891514f242b8a78735404fbbcc19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ad0891514f242b8a78735404fbbcc19, State: Initialized, Role: FOLLOWER
I20260810 01:47:18.466710 20988 consensus_queue.cc:260] T b34233e628544d15bd571d56c67bc734 P 3ad0891514f242b8a78735404fbbcc19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e30d0b090648f29deeb808001b14a7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39543 } } peers { permanent_uuid: "12e99ab08944444b8ea106fa5aa38769" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 46775 } } peers { permanent_uuid: "3ad0891514f242b8a78735404fbbcc19" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45485 } }
I20260810 01:47:18.467160 20988 ts_tablet_manager.cc:1428] T b34233e628544d15bd571d56c67bc734 P 3ad0891514f242b8a78735404fbbcc19: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260810 01:47:18.467875 20945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.467596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.9.77.129:44695)
0810 01:47:18.467711 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:18.467787 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 01:47:18.468932 20379 heartbeater.cc:503] Master 127.9.77.189:38111 was elected leader, sending a full tablet report...
I20260810 01:47:18.469450 20945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.469056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.9.77.129:57473)
0810 01:47:18.469309 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:18.469365 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
I20260810 01:47:18.760589 20986 raft_consensus.cc:482] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:47:18.760828 20986 raft_consensus.cc:504] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e30d0b090648f29deeb808001b14a7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39543 } } peers { permanent_uuid: "12e99ab08944444b8ea106fa5aa38769" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 46775 } } peers { permanent_uuid: "3ad0891514f242b8a78735404fbbcc19" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45485 } }
I20260810 01:47:18.762336 20986 leader_election.cc:288] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49e30d0b090648f29deeb808001b14a7 (127.9.77.131:39543), 3ad0891514f242b8a78735404fbbcc19 (127.9.77.129:45485)
I20260810 01:47:18.766656 20946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.762652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:45485 (local address 127.9.77.130:40893)
0810 01:47:18.762809 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:18.762828 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:18.762980 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:18.763315 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:18.763319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:18.763344 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:18.763484 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:18.763491 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:18.764433 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:18.764439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:18.765412 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:18.765422 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:18.766158 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:18.766162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:18.766273 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:18.766296 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 01:47:18.766392 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 01:47:18.766480 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":22}
I20260810 01:47:18.766656 20977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.762836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:40893 (local address 127.9.77.129:45485)
0810 01:47:18.763013 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:18.763019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:18.763039 (+ 20us) server_negotiation.cc:407] Connection header received
0810 01:47:18.763116 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:18.763119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:18.763175 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:18.763284 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:18.763623 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:18.764308 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:18.765567 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:18.765988 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:18.766144 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:18.766218 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:18.766413 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 01:47:18.766508 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260810 01:47:18.767159 20968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.762805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:54133 (local address 127.9.77.131:39543)
0810 01:47:18.762996 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:18.763002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:18.763489 (+ 487us) server_negotiation.cc:407] Connection header received
0810 01:47:18.763655 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:18.763661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:18.763720 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:18.763810 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:18.764163 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:18.764755 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:18.766026 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:18.766474 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:18.766614 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:18.766677 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:18.766895 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 01:47:18.766992 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 01:47:18.767479 20308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b34233e628544d15bd571d56c67bc734" candidate_uuid: "12e99ab08944444b8ea106fa5aa38769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad0891514f242b8a78735404fbbcc19" is_pre_election: true
I20260810 01:47:18.767602 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.762655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:39543 (local address 127.9.77.130:54133)
0810 01:47:18.763382 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:18.763395 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:18.763525 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:18.763823 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:18.763826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:18.763842 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:18.764048 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:18.764059 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:18.764896 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:18.764902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:18.765899 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:18.765909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:18.766604 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:18.766608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:18.766720 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:18.766749 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 01:47:18.767373 (+ 624us) client_negotiation.cc:240] Negotiation successful
0810 01:47:18.767476 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":226,"thread_start_us":229,"threads_started":1}
I20260810 01:47:18.767828 20308 raft_consensus.cc:2454] T b34233e628544d15bd571d56c67bc734 P 3ad0891514f242b8a78735404fbbcc19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e99ab08944444b8ea106fa5aa38769 in term 0.
I20260810 01:47:18.768250 20732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b34233e628544d15bd571d56c67bc734" candidate_uuid: "12e99ab08944444b8ea106fa5aa38769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e30d0b090648f29deeb808001b14a7" is_pre_election: true
I20260810 01:47:18.768435 20732 raft_consensus.cc:2454] T b34233e628544d15bd571d56c67bc734 P 49e30d0b090648f29deeb808001b14a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12e99ab08944444b8ea106fa5aa38769 in term 0.
I20260810 01:47:18.768455 20420 leader_election.cc:302] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e99ab08944444b8ea106fa5aa38769, 3ad0891514f242b8a78735404fbbcc19; no voters:
I20260810 01:47:18.768858 20986 raft_consensus.cc:2781] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:47:18.768931 20986 raft_consensus.cc:482] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:47:18.768970 20986 raft_consensus.cc:3037] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:18.773530 20986 raft_consensus.cc:504] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e30d0b090648f29deeb808001b14a7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39543 } } peers { permanent_uuid: "12e99ab08944444b8ea106fa5aa38769" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 46775 } } peers { permanent_uuid: "3ad0891514f242b8a78735404fbbcc19" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45485 } }
I20260810 01:47:18.774068 20986 leader_election.cc:288] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769 [CANDIDATE]: Term 1 election: Requested vote from peers 49e30d0b090648f29deeb808001b14a7 (127.9.77.131:39543), 3ad0891514f242b8a78735404fbbcc19 (127.9.77.129:45485)
I20260810 01:47:18.774636 20732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b34233e628544d15bd571d56c67bc734" candidate_uuid: "12e99ab08944444b8ea106fa5aa38769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e30d0b090648f29deeb808001b14a7"
I20260810 01:47:18.774648 20308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b34233e628544d15bd571d56c67bc734" candidate_uuid: "12e99ab08944444b8ea106fa5aa38769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad0891514f242b8a78735404fbbcc19"
I20260810 01:47:18.774822 20732 raft_consensus.cc:3037] T b34233e628544d15bd571d56c67bc734 P 49e30d0b090648f29deeb808001b14a7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:18.774838 20308 raft_consensus.cc:3037] T b34233e628544d15bd571d56c67bc734 P 3ad0891514f242b8a78735404fbbcc19 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:18.776010 20732 raft_consensus.cc:2454] T b34233e628544d15bd571d56c67bc734 P 49e30d0b090648f29deeb808001b14a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e99ab08944444b8ea106fa5aa38769 in term 1.
I20260810 01:47:18.776247 20308 raft_consensus.cc:2454] T b34233e628544d15bd571d56c67bc734 P 3ad0891514f242b8a78735404fbbcc19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12e99ab08944444b8ea106fa5aa38769 in term 1.
I20260810 01:47:18.776491 20423 leader_election.cc:302] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e99ab08944444b8ea106fa5aa38769, 49e30d0b090648f29deeb808001b14a7; no voters:
I20260810 01:47:18.776722 20986 raft_consensus.cc:2781] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:47:18.777040 20986 raft_consensus.cc:686] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769 [term 1 LEADER]: Becoming Leader. State: Replica: 12e99ab08944444b8ea106fa5aa38769, State: Running, Role: LEADER
I20260810 01:47:18.777438 20986 consensus_queue.cc:237] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e30d0b090648f29deeb808001b14a7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39543 } } peers { permanent_uuid: "12e99ab08944444b8ea106fa5aa38769" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 46775 } } peers { permanent_uuid: "3ad0891514f242b8a78735404fbbcc19" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45485 } }
I20260810 01:47:18.778029 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.777755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.9.77.130:56669)
0810 01:47:18.777874 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:18.777948 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 01:47:18.780651 19803 catalog_manager.cc:5495] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12e99ab08944444b8ea106fa5aa38769 (127.9.77.130). New cstate: current_term: 1 leader_uuid: "12e99ab08944444b8ea106fa5aa38769" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e30d0b090648f29deeb808001b14a7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12e99ab08944444b8ea106fa5aa38769" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 46775 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ad0891514f242b8a78735404fbbcc19" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45485 } health_report { overall_health: UNKNOWN } } }
I20260810 01:47:18.819801 20975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.815464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40934 (local address 127.9.77.130:46775)
0810 01:47:18.815659 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:18.815665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:18.815689 (+ 24us) server_negotiation.cc:407] Connection header received
0810 01:47:18.815801 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:18.815803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:18.815900 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:18.816072 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:47:18.816397 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:18.816987 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:18.817945 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:18.818188 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:18.818321 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:18.818651 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:47:18.819330 (+ 679us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:47:18.819485 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:18.819523 (+ 38us) server_negotiation.cc:299] Negotiation successful
0810 01:47:18.819598 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 01:47:18.823788 20732 raft_consensus.cc:1261] T b34233e628544d15bd571d56c67bc734 P 49e30d0b090648f29deeb808001b14a7 [term 1 FOLLOWER]: Refusing update from remote peer 12e99ab08944444b8ea106fa5aa38769: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:47:18.824265 20308 raft_consensus.cc:1261] T b34233e628544d15bd571d56c67bc734 P 3ad0891514f242b8a78735404fbbcc19 [term 1 FOLLOWER]: Refusing update from remote peer 12e99ab08944444b8ea106fa5aa38769: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:47:18.824532 20993 consensus_queue.cc:1035] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e30d0b090648f29deeb808001b14a7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:47:18.825030 20986 consensus_queue.cc:1035] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ad0891514f242b8a78735404fbbcc19" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:47:18.828755 20945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.827975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.9.77.129:55183)
0810 01:47:18.828602 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:18.828666 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":550}
I20260810 01:47:18.830072 20948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.828301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.9.77.131:52635)
0810 01:47:18.829892 (+ 1591us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:18.829965 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1500}
I20260810 01:47:18.832799 20945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.832379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.9.77.129:60229)
0810 01:47:18.832644 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:18.832704 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
I20260810 01:47:18.832856 21003 mvcc.cc:205] Tried to move back new op lower bound from 7316793093415165952 to 7316793093236264960. Current Snapshot: MvccSnapshot[applied={T|T < 7316793093415165952}]
I20260810 01:47:18.833354 20948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.833083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.9.77.131:41467)
0810 01:47:18.833230 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:18.833283 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 01:47:18.834553 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.834081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.9.77.130:53041)
0810 01:47:18.834381 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:18.834455 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226}
I20260810 01:47:18.835621 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.835373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33329 (local address 127.9.77.130:53221)
0810 01:47:18.835488 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:18.835551 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 01:47:18.843911 19803 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:47:18.844195 19803 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:47:18.850169 19803 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3e994dc2a3d412ea8f13f43c39bc783: Sending DeleteTablet for 3 replicas of tablet b34233e628544d15bd571d56c67bc734
I20260810 01:47:18.851189 20493 tablet_service.cc:1514] Processing DeleteTablet for tablet b34233e628544d15bd571d56c67bc734 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:47:18 UTC) from {username='oryx', principal='oryx/127.9.77.189@KRBTEST.COM'} at 127.0.0.1:40928
I20260810 01:47:18.851197 20702 tablet_service.cc:1514] Processing DeleteTablet for tablet b34233e628544d15bd571d56c67bc734 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:47:18 UTC) from {username='oryx', principal='oryx/127.9.77.189@KRBTEST.COM'} at 127.0.0.1:43220
I20260810 01:47:18.851424 20276 tablet_service.cc:1514] Processing DeleteTablet for tablet b34233e628544d15bd571d56c67bc734 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:47:18 UTC) from {username='oryx', principal='oryx/127.9.77.189@KRBTEST.COM'} at 127.0.0.1:55378
I20260810 01:47:18.851931 21016 tablet_replica.cc:330] T b34233e628544d15bd571d56c67bc734 P 3ad0891514f242b8a78735404fbbcc19: stopping tablet replica
I20260810 01:47:18.852145 21016 raft_consensus.cc:2229] T b34233e628544d15bd571d56c67bc734 P 3ad0891514f242b8a78735404fbbcc19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:47:18.852303 21014 tablet_replica.cc:330] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769: stopping tablet replica
I20260810 01:47:18.852347 21016 raft_consensus.cc:2258] T b34233e628544d15bd571d56c67bc734 P 3ad0891514f242b8a78735404fbbcc19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:47:18.852448 21014 raft_consensus.cc:2229] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769 [term 1 LEADER]: Raft consensus shutting down.
I20260810 01:47:18.852860 21014 raft_consensus.cc:2258] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:47:18.853053 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 20040
I20260810 01:47:18.856846 21013 tablet_replica.cc:330] T b34233e628544d15bd571d56c67bc734 P 49e30d0b090648f29deeb808001b14a7: stopping tablet replica
I20260810 01:47:18.857033 21013 raft_consensus.cc:2229] T b34233e628544d15bd571d56c67bc734 P 49e30d0b090648f29deeb808001b14a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:47:18.857200 21013 raft_consensus.cc:2258] T b34233e628544d15bd571d56c67bc734 P 49e30d0b090648f29deeb808001b14a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:47:18.860057 21013 ts_tablet_manager.cc:1905] T b34233e628544d15bd571d56c67bc734 P 49e30d0b090648f29deeb808001b14a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:47:18.863277 21013 ts_tablet_manager.cc:1918] T b34233e628544d15bd571d56c67bc734 P 49e30d0b090648f29deeb808001b14a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:47:18.863379 21013 log.cc:1196] T b34233e628544d15bd571d56c67bc734 P 49e30d0b090648f29deeb808001b14a7: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-2/wal/wals/b34233e628544d15bd571d56c67bc734
I20260810 01:47:18.863682 21013 ts_tablet_manager.cc:1939] T b34233e628544d15bd571d56c67bc734 P 49e30d0b090648f29deeb808001b14a7: Deleting consensus metadata
I20260810 01:47:18.864915 21014 ts_tablet_manager.cc:1905] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:47:18.867940 21014 ts_tablet_manager.cc:1918] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:47:18.868062 21014 log.cc:1196] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786326392877349-9526-0/minicluster-data/ts-1/wal/wals/b34233e628544d15bd571d56c67bc734
I20260810 01:47:18.868363 21014 ts_tablet_manager.cc:1939] T b34233e628544d15bd571d56c67bc734 P 12e99ab08944444b8ea106fa5aa38769: Deleting consensus metadata
I20260810 01:47:18.869410 19786 catalog_manager.cc:4841] TS 12e99ab08944444b8ea106fa5aa38769 (127.9.77.130:46775): tablet b34233e628544d15bd571d56c67bc734 (table test-table [id=8559a9669bd64a80b06e250b547c93c4]) successfully deleted
I20260810 01:47:18.869848 19784 catalog_manager.cc:4841] TS 49e30d0b090648f29deeb808001b14a7 (127.9.77.131:39543): tablet b34233e628544d15bd571d56c67bc734 (table test-table [id=8559a9669bd64a80b06e250b547c93c4]) successfully deleted
I20260810 01:47:18.877410 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 20394
I20260810 01:47:18.877604 20976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:18.877230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:45485 (local address 127.0.0.1:55392)
0810 01:47:18.877391 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:18.877466 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.129:45485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 01:47:18.877887 19781 catalog_manager.cc:4584] TS 3ad0891514f242b8a78735404fbbcc19 (127.9.77.129:45485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b34233e628544d15bd571d56c67bc734: Network error: Client connection negotiation failed: client connection to 127.9.77.129:45485: connect: Connection refused (error 111)
I20260810 01:47:18.889865 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 20604
I20260810 01:47:18.906455 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 19581
I20260810 01:47:18.928973 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 19739
2026-08-10T01:47:18Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8222 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:47:19 dist-test-slave-kvfs krb5kdc[21019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:47:19 dist-test-slave-kvfs krb5kdc[21019](info): set up 2 sockets
Aug 10 01:47:19 dist-test-slave-kvfs krb5kdc[21019](info): commencing operation
krb5kdc: starting...
W20260810 01:47:20.284493 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.315s user 0.003s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:47:20 dist-test-slave-kvfs krb5kdc[21019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:47:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:47:20Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.190.keytab.
Entry for principal oryx/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.190.keytab.
I20260810 01:47:20.440761 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:46517
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.190.keytab
--principal=oryx/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:42961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:46517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:20.579149 21044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:20.579551 21044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:20.579599 21044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:20.584223 21044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:47:20.584336 21044 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 01:47:20.584355 21044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:20.584381 21044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:20.584400 21044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:47:20.584416 21044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:47:20.590844 21044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:42961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:46517
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.190.keytab
--principal=oryx/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:46517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:20.592588 21044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:20.594085 21044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 01:47:20.606108 21044 server_base.cc:1034] running on GCE node
W20260810 01:47:20.606015 21056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:20.606082 21054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:20.606026 21060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:20.607187 21044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:20.608922 21044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:20.610126 21044 hybrid_clock.cc:648] HybridClock initialized: now 1786326440610082 us; error 60 us; skew 500 ppm
Aug 10 01:47:20 dist-test-slave-kvfs krb5kdc[21019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326440, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:20.613787 21044 init.cc:377] Logged in from keytab as oryx/127.9.77.190@KRBTEST.COM (short username oryx)
I20260810 01:47:20.614892 21044 webserver.cc:460] Webserver started at http://127.9.77.190:44403/ using document root <none> and password file <none>
I20260810 01:47:20.615584 21044 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:20.615653 21044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:20.615876 21044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:20.618580 21044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "6fab89cac2fe444da5e6f375fef7e249"
format_stamp: "Formatted at 2026-08-10 01:47:20 on dist-test-slave-kvfs"
server_key: "7306802e436893220c20d0bffac8e684"
server_key_iv: "7d28974880986b4ce5204a9ccd8b8674"
server_key_version: "encryptionkey@0"
I20260810 01:47:20.619225 21044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "6fab89cac2fe444da5e6f375fef7e249"
format_stamp: "Formatted at 2026-08-10 01:47:20 on dist-test-slave-kvfs"
server_key: "7306802e436893220c20d0bffac8e684"
server_key_iv: "7d28974880986b4ce5204a9ccd8b8674"
server_key_version: "encryptionkey@0"
I20260810 01:47:20.623884 21044 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 01:47:20.627022 21071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:20.628728 21044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:47:20.628895 21044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "6fab89cac2fe444da5e6f375fef7e249"
format_stamp: "Formatted at 2026-08-10 01:47:20 on dist-test-slave-kvfs"
server_key: "7306802e436893220c20d0bffac8e684"
server_key_iv: "7d28974880986b4ce5204a9ccd8b8674"
server_key_version: "encryptionkey@0"
I20260810 01:47:20.629060 21044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:20.645794 21044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:20.649746 21044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:20.650028 21044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:20.660164 21044 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:46517
I20260810 01:47:20.660185 21165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:46517 every 8 connection(s)
I20260810 01:47:20.661445 21044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:47:20.665324 21167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:20.671814 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 21044
I20260810 01:47:20.672144 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:47:20.672550 9526 external_mini_cluster.cc:1428] Setting key 592caa046942b908260afa95d0e2ccae
I20260810 01:47:20.673029 21167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fab89cac2fe444da5e6f375fef7e249: Bootstrap starting.
I20260810 01:47:20.676628 21167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fab89cac2fe444da5e6f375fef7e249: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:20.677860 21167 log.cc:824] T 00000000000000000000000000000000 P 6fab89cac2fe444da5e6f375fef7e249: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:20.681007 21167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fab89cac2fe444da5e6f375fef7e249: No bootstrap required, opened a new log
Aug 10 01:47:20 dist-test-slave-kvfs krb5kdc[21019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM
I20260810 01:47:20.686462 21167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fab89cac2fe444da5e6f375fef7e249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fab89cac2fe444da5e6f375fef7e249" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46517 } }
I20260810 01:47:20.686923 21167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fab89cac2fe444da5e6f375fef7e249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:20.686986 21167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fab89cac2fe444da5e6f375fef7e249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fab89cac2fe444da5e6f375fef7e249, State: Initialized, Role: FOLLOWER
I20260810 01:47:20.687719 21167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fab89cac2fe444da5e6f375fef7e249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fab89cac2fe444da5e6f375fef7e249" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46517 } }
I20260810 01:47:20.687934 21167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fab89cac2fe444da5e6f375fef7e249 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:47:20.688043 21167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fab89cac2fe444da5e6f375fef7e249 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:47:20.688172 21167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fab89cac2fe444da5e6f375fef7e249 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:20.689826 21167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fab89cac2fe444da5e6f375fef7e249 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fab89cac2fe444da5e6f375fef7e249" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46517 } }
I20260810 01:47:20.690330 21167 leader_election.cc:302] T 00000000000000000000000000000000 P 6fab89cac2fe444da5e6f375fef7e249 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fab89cac2fe444da5e6f375fef7e249; no voters:
I20260810 01:47:20.690810 21167 leader_election.cc:288] T 00000000000000000000000000000000 P 6fab89cac2fe444da5e6f375fef7e249 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:47:20.690946 21175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fab89cac2fe444da5e6f375fef7e249 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:47:20.691254 21175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fab89cac2fe444da5e6f375fef7e249 [term 1 LEADER]: Becoming Leader. State: Replica: 6fab89cac2fe444da5e6f375fef7e249, State: Running, Role: LEADER
I20260810 01:47:20.691746 21175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fab89cac2fe444da5e6f375fef7e249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fab89cac2fe444da5e6f375fef7e249" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46517 } }
I20260810 01:47:20.692240 21170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:20.674492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33844 (local address 127.9.77.190:46517)
0810 01:47:20.675077 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:20.675101 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:20.675156 (+ 55us) server_negotiation.cc:407] Connection header received
0810 01:47:20.676235 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:20.676266 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:20.676826 (+ 560us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:20.677342 (+ 516us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:20.678280 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:20.679800 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:20.680936 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:20.681737 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:20.681956 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:20.685312 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:20.685321 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:20.685348 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:20.685394 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:20.687812 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:20.688531 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:20.688540 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:20.688547 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:20.688718 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:20.689041 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:20.689045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:20.689046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:20.689747 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:20.689994 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:20.690422 (+ 428us) server_negotiation.cc:299] Negotiation successful
0810 01:47:20.690831 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":164,"threads_started":1}
I20260810 01:47:20.692526 21167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fab89cac2fe444da5e6f375fef7e249 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:47:20.695019 21176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fab89cac2fe444da5e6f375fef7e249 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fab89cac2fe444da5e6f375fef7e249. Latest consensus state: current_term: 1 leader_uuid: "6fab89cac2fe444da5e6f375fef7e249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fab89cac2fe444da5e6f375fef7e249" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46517 } } }
I20260810 01:47:20.695899 21176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fab89cac2fe444da5e6f375fef7e249 [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:20.695202 21177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fab89cac2fe444da5e6f375fef7e249 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fab89cac2fe444da5e6f375fef7e249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fab89cac2fe444da5e6f375fef7e249" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46517 } } }
I20260810 01:47:20.696151 21177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fab89cac2fe444da5e6f375fef7e249 [sys.catalog]: This master's current role is: LEADER
W20260810 01:47:20.698851 21197 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6fab89cac2fe444da5e6f375fef7e249: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 01:47:20.698941 21197 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 01:47:20.699012 21198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:47:20.699905 21198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:47:20.707109 21198 catalog_manager.cc:1347] Generated new cluster ID: 2777c1eecb67437b920969ac134b08e1
I20260810 01:47:20.707207 21198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:47:20.716480 21198 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:47:20.717780 21198 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:47:20.727273 21198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fab89cac2fe444da5e6f375fef7e249: Generated new TSK 0
I20260810 01:47:20.728559 21198 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.77.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.77.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.77.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.129.keytab.
Entry for principal oryx/127.9.77.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.129.keytab.
I20260810 01:47:20.780453 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:0
--local_ip_for_outbound_sockets=127.9.77.129
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:46517
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.129.keytab
--principal=oryx/127.9.77.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:42961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:20.937165 21209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:20.937577 21209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:20.937629 21209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:20.942116 21209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:20.942219 21209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:20.942265 21209 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 01:47:20.942302 21209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:47:20.948694 21209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:42961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.129.keytab
--principal=oryx/127.9.77.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.77.190:46517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:20.950268 21209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:20.951705 21209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:20.964033 21221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:20.964071 21223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:20.964071 21217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:20.964432 21209 server_base.cc:1034] running on GCE node
I20260810 01:47:20.964913 21209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:20.965746 21209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:20.966958 21209 hybrid_clock.cc:648] HybridClock initialized: now 1786326440966936 us; error 38 us; skew 500 ppm
Aug 10 01:47:20 dist-test-slave-kvfs krb5kdc[21019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326440, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:20.970345 21209 init.cc:377] Logged in from keytab as oryx/127.9.77.129@KRBTEST.COM (short username oryx)
I20260810 01:47:20.971446 21209 webserver.cc:460] Webserver started at http://127.9.77.129:44725/ using document root <none> and password file <none>
I20260810 01:47:20.972229 21209 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:20.972332 21209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:20.972625 21209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:20.975265 21209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-0/data/instance:
uuid: "a505fe46057f4a7da2c7ce013cb68638"
format_stamp: "Formatted at 2026-08-10 01:47:20 on dist-test-slave-kvfs"
server_key: "694ba4589620d6ca9e2c60c158c8f6f7"
server_key_iv: "06b4ffc824ae40a398aca8e41ede030b"
server_key_version: "encryptionkey@0"
I20260810 01:47:20.975965 21209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance:
uuid: "a505fe46057f4a7da2c7ce013cb68638"
format_stamp: "Formatted at 2026-08-10 01:47:20 on dist-test-slave-kvfs"
server_key: "694ba4589620d6ca9e2c60c158c8f6f7"
server_key_iv: "06b4ffc824ae40a398aca8e41ede030b"
server_key_version: "encryptionkey@0"
I20260810 01:47:20.980782 21209 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 01:47:20.983873 21233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:20.985699 21209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:47:20.985826 21209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "a505fe46057f4a7da2c7ce013cb68638"
format_stamp: "Formatted at 2026-08-10 01:47:20 on dist-test-slave-kvfs"
server_key: "694ba4589620d6ca9e2c60c158c8f6f7"
server_key_iv: "06b4ffc824ae40a398aca8e41ede030b"
server_key_version: "encryptionkey@0"
I20260810 01:47:20.985945 21209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:21.015045 21209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:21.019006 21209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:21.019217 21209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:21.021067 21209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:21.021173 21209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:21.021266 21209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:21.021315 21209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 01:47:21 dist-test-slave-kvfs krb5kdc[21019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326440, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM
I20260810 01:47:21.034772 21209 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:41641
I20260810 01:47:21.034905 21419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:41641 every 8 connection(s)
I20260810 01:47:21.036082 21209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:47:21.037577 21170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:21.022273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:36135 (local address 127.9.77.190:46517)
0810 01:47:21.022488 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:21.022494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:21.023573 (+ 1079us) server_negotiation.cc:407] Connection header received
0810 01:47:21.025261 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:21.025265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:21.025348 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:21.025469 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:21.027159 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.027810 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.029178 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.029482 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.029613 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:21.032891 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:21.032895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:21.032898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:21.032944 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:21.034915 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:21.035527 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:21.035534 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:21.035536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:21.035649 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:21.036125 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:21.036128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:21.036130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:21.036416 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:21.036558 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:21.037271 (+ 713us) server_negotiation.cc:299] Negotiation successful
0810 01:47:21.037411 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 01:47:21.038586 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:21.022738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46517 (local address 127.9.77.129:36135)
0810 01:47:21.023329 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:21.023419 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:21.024910 (+ 1491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:21.025685 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:21.025693 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:21.026167 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:21.026984 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:21.027005 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.027960 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.027964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:21.029033 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:21.029044 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.029681 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.029685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:21.029964 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:21.030708 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:21.030735 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:21.032682 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:21.035100 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:21.035105 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:21.035119 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:21.035403 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:21.035773 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:21.035777 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:21.035779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:21.035976 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:21.036575 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:21.036586 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:21.036943 (+ 357us) client_negotiation.cc:769] Sending connection context
0810 01:47:21.037190 (+ 247us) client_negotiation.cc:240] Negotiation successful
0810 01:47:21.037555 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":118,"threads_started":1}
I20260810 01:47:21.040220 21420 heartbeater.cc:347] Connected to a master server at 127.9.77.190:46517
I20260810 01:47:21.040390 21420 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:21.040994 21420 heartbeater.cc:511] Master 127.9.77.190:46517 requested a full tablet report, sending...
I20260810 01:47:21.042498 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 21209
I20260810 01:47:21.042659 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance
I20260810 01:47:21.043141 9526 external_mini_cluster.cc:1428] Setting key 43618e72bc0afce0b4064aeb72e2dcdd
I20260810 01:47:21.043136 21092 ts_manager.cc:194] Registered new tserver with Master: a505fe46057f4a7da2c7ce013cb68638 (127.9.77.129:41641)
I20260810 01:47:21.044940 21092 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.77.129@KRBTEST.COM'} at 127.9.77.129:36135
I20260810 01:47:21.051811 21170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:21.043072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:49769 (local address 127.9.77.190:46517)
0810 01:47:21.043206 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:21.043211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:21.043511 (+ 300us) server_negotiation.cc:407] Connection header received
0810 01:47:21.043799 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:21.043802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:21.043851 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:21.043950 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:21.044492 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.045202 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.046398 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.046797 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.046928 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:21.048411 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:21.048414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:21.048416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:21.048441 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:21.049986 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:21.050538 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:21.050541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:21.050543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:21.050630 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:21.050951 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:21.050954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:21.050955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:21.051141 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:21.051221 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:21.051550 (+ 329us) server_negotiation.cc:299] Negotiation successful
0810 01:47:21.051667 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 01:47:21.052012 21427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:21.042990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46517 (local address 127.9.77.129:49769)
0810 01:47:21.043379 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:21.043396 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:21.043525 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:21.043970 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:21.043973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:21.044246 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:21.044366 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:21.044374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.045361 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.045365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:21.046252 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:21.046264 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.046938 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.046943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:21.047036 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:21.047725 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:21.047741 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:21.048267 (+ 526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:21.050114 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:21.050120 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:21.050125 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:21.050418 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:21.050724 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:21.050730 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:21.050735 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:21.050850 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:21.051237 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:21.051243 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:21.051371 (+ 128us) client_negotiation.cc:769] Sending connection context
0810 01:47:21.051499 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 01:47:21.051700 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for oryx/127.9.77.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.77.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.77.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.130.keytab.
Entry for principal oryx/127.9.77.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.130.keytab.
I20260810 01:47:21.079784 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:0
--local_ip_for_outbound_sockets=127.9.77.130
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:46517
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.130.keytab
--principal=oryx/127.9.77.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:42961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:21.218765 21433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:21.219204 21433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:21.219277 21433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:21.224285 21433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:21.224365 21433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:21.224409 21433 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 01:47:21.224447 21433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:47:21.230677 21433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:42961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.130.keytab
--principal=oryx/127.9.77.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.77.190:46517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:21.232237 21433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:21.233678 21433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:21.247323 21440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:21.247323 21442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:21.247349 21438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:21.247929 21433 server_base.cc:1034] running on GCE node
I20260810 01:47:21.248502 21433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:21.249358 21433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:21.250576 21433 hybrid_clock.cc:648] HybridClock initialized: now 1786326441250545 us; error 49 us; skew 500 ppm
Aug 10 01:47:21 dist-test-slave-kvfs krb5kdc[21019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326441, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:21.253988 21433 init.cc:377] Logged in from keytab as oryx/127.9.77.130@KRBTEST.COM (short username oryx)
I20260810 01:47:21.255170 21433 webserver.cc:460] Webserver started at http://127.9.77.130:36063/ using document root <none> and password file <none>
I20260810 01:47:21.255980 21433 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:21.256102 21433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:21.256413 21433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:21.259060 21433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-1/data/instance:
uuid: "b89e2268ec7242c992c04fc4f68afdeb"
format_stamp: "Formatted at 2026-08-10 01:47:21 on dist-test-slave-kvfs"
server_key: "0d4a1ded4a6b358187b30fd2686593f8"
server_key_iv: "608c444fd87723e76fa47e006e1df3c0"
server_key_version: "encryptionkey@0"
I20260810 01:47:21.259663 21433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance:
uuid: "b89e2268ec7242c992c04fc4f68afdeb"
format_stamp: "Formatted at 2026-08-10 01:47:21 on dist-test-slave-kvfs"
server_key: "0d4a1ded4a6b358187b30fd2686593f8"
server_key_iv: "608c444fd87723e76fa47e006e1df3c0"
server_key_version: "encryptionkey@0"
I20260810 01:47:21.264365 21433 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 01:47:21.267311 21453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:21.269011 21433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:47:21.269160 21433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "b89e2268ec7242c992c04fc4f68afdeb"
format_stamp: "Formatted at 2026-08-10 01:47:21 on dist-test-slave-kvfs"
server_key: "0d4a1ded4a6b358187b30fd2686593f8"
server_key_iv: "608c444fd87723e76fa47e006e1df3c0"
server_key_version: "encryptionkey@0"
I20260810 01:47:21.269302 21433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:21.282181 21433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:21.286360 21433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:21.286631 21433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:21.288663 21433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:21.288746 21433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:21.288830 21433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:21.288882 21433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 01:47:21 dist-test-slave-kvfs krb5kdc[21019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326441, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM
I20260810 01:47:21.302126 21433 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:44931
I20260810 01:47:21.302150 21637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:44931 every 8 connection(s)
I20260810 01:47:21.303277 21433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:47:21.304994 21170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:21.289649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:38417 (local address 127.9.77.190:46517)
0810 01:47:21.289833 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:21.289838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:21.291243 (+ 1405us) server_negotiation.cc:407] Connection header received
0810 01:47:21.292989 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:21.292995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:21.293084 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:21.293187 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:21.294876 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.295599 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.297178 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.297533 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.297661 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:21.300676 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:21.300680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:21.300683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:21.300722 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:21.302460 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:21.303004 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:21.303008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:21.303010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:21.303103 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:21.303524 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:21.303527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:21.303529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:21.303830 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:21.303947 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:21.304647 (+ 700us) server_negotiation.cc:299] Negotiation successful
0810 01:47:21.304801 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 01:47:21.305966 21466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:21.290319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46517 (local address 127.9.77.130:38417)
0810 01:47:21.291009 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:21.291106 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:21.292614 (+ 1508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:21.293511 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:21.293522 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:21.294071 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:21.294714 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:21.294730 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.295736 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.295740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:21.297052 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:21.297062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.297722 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.297726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:21.297936 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:21.298722 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:21.298741 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:21.300477 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:21.302582 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:21.302589 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:21.302607 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:21.302891 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:21.303200 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:21.303204 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:21.303206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:21.303409 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:21.304003 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:21.304010 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:21.304377 (+ 367us) client_negotiation.cc:769] Sending connection context
0810 01:47:21.304585 (+ 208us) client_negotiation.cc:240] Negotiation successful
0810 01:47:21.304972 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":129,"threads_started":1}
I20260810 01:47:21.307384 21638 heartbeater.cc:347] Connected to a master server at 127.9.77.190:46517
I20260810 01:47:21.307670 21638 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:21.308400 21638 heartbeater.cc:511] Master 127.9.77.190:46517 requested a full tablet report, sending...
I20260810 01:47:21.309819 21094 ts_manager.cc:194] Registered new tserver with Master: b89e2268ec7242c992c04fc4f68afdeb (127.9.77.130:44931)
I20260810 01:47:21.309929 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 21433
I20260810 01:47:21.310066 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance
I20260810 01:47:21.310451 9526 external_mini_cluster.cc:1428] Setting key 276037c760411fabad9925f8424fb9d2
I20260810 01:47:21.310755 21094 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.77.130@KRBTEST.COM'} at 127.9.77.130:38417
I20260810 01:47:21.318312 21649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:21.310461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46517 (local address 127.9.77.130:51419)
0810 01:47:21.310831 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:21.310864 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:21.311019 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:21.311327 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:21.311330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:21.311571 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:21.311707 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:21.311714 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.312606 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.312610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:21.313200 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:21.313209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.313693 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.313697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:21.313797 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:21.314440 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:21.314453 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:21.314811 (+ 358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:21.316595 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:21.316600 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:21.316603 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:21.316824 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:21.317129 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:21.317133 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:21.317135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:21.317232 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:21.317705 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:21.317708 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:21.317816 (+ 108us) client_negotiation.cc:769] Sending connection context
0810 01:47:21.317932 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 01:47:21.318110 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":121,"threads_started":1}
I20260810 01:47:21.318302 21170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:21.310509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:51419 (local address 127.9.77.190:46517)
0810 01:47:21.310644 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:21.310648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:21.310942 (+ 294us) server_negotiation.cc:407] Connection header received
0810 01:47:21.311146 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:21.311151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:21.311211 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:21.311301 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:21.311828 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.312467 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.313332 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.313572 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.313688 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:21.314957 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:21.314961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:21.314964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:21.314996 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:21.316473 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:21.316942 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:21.316951 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:21.316952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:21.317036 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:21.317328 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:21.317331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:21.317333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:21.317601 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:21.317720 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:21.317991 (+ 271us) server_negotiation.cc:299] Negotiation successful
0810 01:47:21.318112 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for oryx/127.9.77.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.77.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.77.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.131.keytab.
Entry for principal oryx/127.9.77.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.131.keytab.
I20260810 01:47:21.348440 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:0
--local_ip_for_outbound_sockets=127.9.77.131
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:46517
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.131.keytab
--principal=oryx/127.9.77.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:42961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:21.488327 21653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:21.488682 21653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:21.488732 21653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:21.493474 21653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:21.493572 21653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:21.493651 21653 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 01:47:21.493728 21653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:47:21.499914 21653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:42961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.131.keytab
--principal=oryx/127.9.77.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.77.190:46517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:21.501471 21653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:21.502833 21653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:21.514957 21661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:21.514976 21664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:21.515113 21653 server_base.cc:1034] running on GCE node
W20260810 01:47:21.514981 21660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:21.515794 21653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:21.516654 21653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:21.517824 21653 hybrid_clock.cc:648] HybridClock initialized: now 1786326441517810 us; error 27 us; skew 500 ppm
Aug 10 01:47:21 dist-test-slave-kvfs krb5kdc[21019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326441, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:21.521092 21653 init.cc:377] Logged in from keytab as oryx/127.9.77.131@KRBTEST.COM (short username oryx)
I20260810 01:47:21.522174 21653 webserver.cc:460] Webserver started at http://127.9.77.131:34035/ using document root <none> and password file <none>
I20260810 01:47:21.522872 21653 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:21.522943 21653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:21.523206 21653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:21.525750 21653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-2/data/instance:
uuid: "c86a959ed6f54447bf5c79b8dc35f9f2"
format_stamp: "Formatted at 2026-08-10 01:47:21 on dist-test-slave-kvfs"
server_key: "564d49274f27dd5b845016c2c54ce946"
server_key_iv: "99f9dfba7fabc45445d5f06b4388a704"
server_key_version: "encryptionkey@0"
I20260810 01:47:21.526402 21653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance:
uuid: "c86a959ed6f54447bf5c79b8dc35f9f2"
format_stamp: "Formatted at 2026-08-10 01:47:21 on dist-test-slave-kvfs"
server_key: "564d49274f27dd5b845016c2c54ce946"
server_key_iv: "99f9dfba7fabc45445d5f06b4388a704"
server_key_version: "encryptionkey@0"
I20260810 01:47:21.531090 21653 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 01:47:21.534358 21675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:21.536013 21653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 01:47:21.536226 21653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "c86a959ed6f54447bf5c79b8dc35f9f2"
format_stamp: "Formatted at 2026-08-10 01:47:21 on dist-test-slave-kvfs"
server_key: "564d49274f27dd5b845016c2c54ce946"
server_key_iv: "99f9dfba7fabc45445d5f06b4388a704"
server_key_version: "encryptionkey@0"
I20260810 01:47:21.536372 21653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:21.549883 21653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:21.553896 21653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:21.554157 21653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:21.556186 21653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:21.556267 21653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:21.556352 21653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:21.556389 21653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 01:47:21 dist-test-slave-kvfs krb5kdc[21019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326441, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM
I20260810 01:47:21.569679 21653 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:35369
I20260810 01:47:21.570140 21865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:35369 every 8 connection(s)
I20260810 01:47:21.571117 21653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:47:21.574149 21170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:21.557370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:46601 (local address 127.9.77.190:46517)
0810 01:47:21.557548 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:21.557552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:21.558870 (+ 1318us) server_negotiation.cc:407] Connection header received
0810 01:47:21.560276 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:21.560280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:21.560393 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:21.560494 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:21.562620 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.563244 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.564898 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.565149 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.565284 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:21.569416 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:21.569419 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:21.569421 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:21.569474 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:21.571427 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:21.572181 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:21.572185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:21.572187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:21.572296 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:21.572700 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:21.572703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:21.572705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:21.572975 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:21.573098 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:21.573758 (+ 660us) server_negotiation.cc:299] Negotiation successful
0810 01:47:21.573941 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 01:47:21.575218 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:21.557993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46517 (local address 127.9.77.131:46601)
0810 01:47:21.558659 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:21.558713 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:21.559936 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:21.560803 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:21.560817 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:21.561429 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:21.562394 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:21.562428 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.563401 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.563405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:21.564742 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:21.564761 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.565351 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.565356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:21.565564 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:21.566896 (+ 1332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:21.566916 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:21.569084 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:21.571570 (+ 2486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:21.571577 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:21.571597 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:21.572003 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:21.572415 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:21.572419 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:21.572421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:21.572584 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:21.573092 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:21.573099 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:21.573471 (+ 372us) client_negotiation.cc:769] Sending connection context
0810 01:47:21.573686 (+ 215us) client_negotiation.cc:240] Negotiation successful
0810 01:47:21.574098 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":188,"threads_started":1}
I20260810 01:47:21.576737 21866 heartbeater.cc:347] Connected to a master server at 127.9.77.190:46517
I20260810 01:47:21.576947 21866 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:21.577589 21866 heartbeater.cc:511] Master 127.9.77.190:46517 requested a full tablet report, sending...
I20260810 01:47:21.578576 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 21653
I20260810 01:47:21.578748 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance
I20260810 01:47:21.578935 21092 ts_manager.cc:194] Registered new tserver with Master: c86a959ed6f54447bf5c79b8dc35f9f2 (127.9.77.131:35369)
I20260810 01:47:21.579160 9526 external_mini_cluster.cc:1428] Setting key 7c67630d650df771ae7a3ce8ef66c36c
I20260810 01:47:21.579861 21092 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.77.131@KRBTEST.COM'} at 127.9.77.131:46601
I20260810 01:47:21.582293 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 01:47:21.588713 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:21.579766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46517 (local address 127.9.77.131:50415)
0810 01:47:21.580216 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:21.580230 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:21.580423 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:21.580835 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:21.580838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:21.581116 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:21.581286 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:21.581297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.582142 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.582145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:21.583012 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:21.583025 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.583529 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.583535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:21.583664 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:21.584380 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:21.584402 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:21.584972 (+ 570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:21.586974 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:21.586980 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:21.586983 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:21.587208 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:21.587534 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:21.587539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:21.587541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:21.587640 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:21.588094 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:21.588097 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:21.588204 (+ 107us) client_negotiation.cc:769] Sending connection context
0810 01:47:21.588314 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 01:47:21.588503 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":116,"threads_started":1}
I20260810 01:47:21.588708 21170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:21.579851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:50415 (local address 127.9.77.190:46517)
0810 01:47:21.580055 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:21.580063 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:21.580331 (+ 268us) server_negotiation.cc:407] Connection header received
0810 01:47:21.580546 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:21.580549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:21.580611 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:21.580670 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:21.581397 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.581987 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.583161 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.583393 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.583490 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:21.585189 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:21.585192 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:21.585195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:21.585219 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:21.586825 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:21.587317 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:21.587320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:21.587322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:21.587420 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:21.587735 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:21.587739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:21.587741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:21.587956 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:21.588094 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:21.588384 (+ 290us) server_negotiation.cc:299] Negotiation successful
0810 01:47:21.588515 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 01:47:21 dist-test-slave-kvfs krb5kdc[21019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326441, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 01:47:21 dist-test-slave-kvfs krb5kdc[21019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:21 dist-test-slave-kvfs krb5kdc[21019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
W20260810 01:47:21.614876 21170 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:21.606887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60330 (local address 127.9.77.190:46517)
0810 01:47:21.607043 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:21.607046 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:21.607062 (+ 16us) server_negotiation.cc:407] Connection header received
0810 01:47:21.607119 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:21.607121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:21.607169 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:21.607246 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:21.608337 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.609044 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.610115 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.610354 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.610445 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:21.614684 (+ 4239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60330: BlockingRecv error: recv got EOF from 127.0.0.1:60330 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 01:47:21 dist-test-slave-kvfs krb5kdc[21019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326441, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM
I20260810 01:47:21.626060 21170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:21.616799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60336 (local address 127.9.77.190:46517)
0810 01:47:21.616953 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:21.616959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:21.617150 (+ 191us) server_negotiation.cc:407] Connection header received
0810 01:47:21.617265 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:21.617267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:21.617318 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:21.617396 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:21.618060 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.618610 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.619462 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.619688 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.619782 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:21.622588 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:21.622591 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:21.622594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:21.622623 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:21.624325 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:21.624769 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:21.624773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:21.624774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:21.624862 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:21.625115 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:21.625117 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:21.625118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:21.625333 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:21.625468 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:21.625698 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 01:47:21.625812 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 01:47:21.629786 21092 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 01:47:21.633760 21092 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:47:21.652357 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:21.646367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:41641 (local address 127.0.0.1:50576)
0810 01:47:21.646643 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:21.646660 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:21.646758 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:21.647789 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:21.647792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:21.647813 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:21.647936 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:21.647942 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.649554 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.649558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:21.650593 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:21.650604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.651683 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.651687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:21.651813 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:21.651840 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 01:47:21.651944 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 01:47:21.652156 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":164,"thread_start_us":149,"threads_started":1}
I20260810 01:47:21.652356 21906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:21.645742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60000 (local address 127.9.77.130:44931)
0810 01:47:21.646660 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:21.646671 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:21.646694 (+ 23us) server_negotiation.cc:407] Connection header received
0810 01:47:21.646772 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:21.646802 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:21.647080 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:21.647303 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:21.647687 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.648819 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.650770 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.651507 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.651642 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:21.651867 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:21.651993 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 01:47:21.652155 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":810,"thread_start_us":145,"threads_started":1}
I20260810 01:47:21.652908 21907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:21.645723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:35369 (local address 127.0.0.1:60672)
0810 01:47:21.646210 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:21.646228 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:21.646338 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:21.648068 (+ 1730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:21.648071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:21.648093 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:21.648228 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:21.648236 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.649810 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.649815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:21.650974 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:21.650983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.652284 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.652287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:21.652398 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:21.652427 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 01:47:21.652710 (+ 283us) client_negotiation.cc:240] Negotiation successful
0810 01:47:21.652786 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":165,"threads_started":1}
I20260810 01:47:21.653095 21905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:21.645549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:44931 (local address 127.0.0.1:60000)
0810 01:47:21.646097 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:21.646160 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:21.646321 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:21.647287 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:21.647295 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:21.647320 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:21.647546 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:21.647556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.648947 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.648954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:21.650618 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:21.650629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.651642 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.651645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:21.651788 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:21.651833 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 01:47:21.651961 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 01:47:21.652158 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":33,"thread_start_us":91,"threads_started":1}
I20260810 01:47:21.653517 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:21.646976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60672 (local address 127.9.77.131:35369)
0810 01:47:21.647536 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:21.647550 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:21.647574 (+ 24us) server_negotiation.cc:407] Connection header received
0810 01:47:21.647661 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:21.647666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:21.647852 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:21.648398 (+ 546us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:21.648571 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.649684 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.651102 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.651892 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.652007 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:21.652222 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:21.653100 (+ 878us) server_negotiation.cc:299] Negotiation successful
0810 01:47:21.653340 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":173,"threads_started":1}
I20260810 01:47:21.653790 21910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:21.646921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50576 (local address 127.9.77.129:41641)
0810 01:47:21.647257 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:21.647265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:21.647292 (+ 27us) server_negotiation.cc:407] Connection header received
0810 01:47:21.647389 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:21.647394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:21.647633 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:21.647792 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:21.648198 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.649395 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.650728 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.651533 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.653287 (+ 1754us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:21.653456 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:21.653523 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 01:47:21.653647 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":100,"threads_started":1}
I20260810 01:47:21.656971 21313 tablet_service.cc:1467] Processing CreateTablet for tablet 0f1a8e78ba774996be13eb46a950cb12 (DEFAULT_TABLE table=test-table [id=a84b873b6c244723a7ca0cc24e6070b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:47:21.656975 21751 tablet_service.cc:1467] Processing CreateTablet for tablet 0f1a8e78ba774996be13eb46a950cb12 (DEFAULT_TABLE table=test-table [id=a84b873b6c244723a7ca0cc24e6070b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:47:21.656978 21534 tablet_service.cc:1467] Processing CreateTablet for tablet 0f1a8e78ba774996be13eb46a950cb12 (DEFAULT_TABLE table=test-table [id=a84b873b6c244723a7ca0cc24e6070b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:47:21.658249 21534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f1a8e78ba774996be13eb46a950cb12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:21.658252 21751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f1a8e78ba774996be13eb46a950cb12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:21.658264 21313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f1a8e78ba774996be13eb46a950cb12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:21.668201 21916 tablet_bootstrap.cc:492] T 0f1a8e78ba774996be13eb46a950cb12 P b89e2268ec7242c992c04fc4f68afdeb: Bootstrap starting.
I20260810 01:47:21.669157 21913 tablet_bootstrap.cc:492] T 0f1a8e78ba774996be13eb46a950cb12 P a505fe46057f4a7da2c7ce013cb68638: Bootstrap starting.
I20260810 01:47:21.671204 21913 tablet_bootstrap.cc:654] T 0f1a8e78ba774996be13eb46a950cb12 P a505fe46057f4a7da2c7ce013cb68638: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:21.671203 21916 tablet_bootstrap.cc:654] T 0f1a8e78ba774996be13eb46a950cb12 P b89e2268ec7242c992c04fc4f68afdeb: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:21.672194 21916 log.cc:824] T 0f1a8e78ba774996be13eb46a950cb12 P b89e2268ec7242c992c04fc4f68afdeb: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:21.672195 21913 log.cc:824] T 0f1a8e78ba774996be13eb46a950cb12 P a505fe46057f4a7da2c7ce013cb68638: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:21.672590 21914 tablet_bootstrap.cc:492] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2: Bootstrap starting.
I20260810 01:47:21.674794 21916 tablet_bootstrap.cc:492] T 0f1a8e78ba774996be13eb46a950cb12 P b89e2268ec7242c992c04fc4f68afdeb: No bootstrap required, opened a new log
I20260810 01:47:21.675086 21916 ts_tablet_manager.cc:1397] T 0f1a8e78ba774996be13eb46a950cb12 P b89e2268ec7242c992c04fc4f68afdeb: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260810 01:47:21.675179 21914 tablet_bootstrap.cc:654] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:21.675927 21914 log.cc:824] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:21.678287 21914 tablet_bootstrap.cc:492] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2: No bootstrap required, opened a new log
I20260810 01:47:21.678519 21914 ts_tablet_manager.cc:1397] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260810 01:47:21.678654 21913 tablet_bootstrap.cc:492] T 0f1a8e78ba774996be13eb46a950cb12 P a505fe46057f4a7da2c7ce013cb68638: No bootstrap required, opened a new log
I20260810 01:47:21.678906 21913 ts_tablet_manager.cc:1397] T 0f1a8e78ba774996be13eb46a950cb12 P a505fe46057f4a7da2c7ce013cb68638: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260810 01:47:21.679534 21916 raft_consensus.cc:348] T 0f1a8e78ba774996be13eb46a950cb12 P b89e2268ec7242c992c04fc4f68afdeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } } peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } }
I20260810 01:47:21.679836 21916 raft_consensus.cc:374] T 0f1a8e78ba774996be13eb46a950cb12 P b89e2268ec7242c992c04fc4f68afdeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:21.679894 21916 raft_consensus.cc:729] T 0f1a8e78ba774996be13eb46a950cb12 P b89e2268ec7242c992c04fc4f68afdeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89e2268ec7242c992c04fc4f68afdeb, State: Initialized, Role: FOLLOWER
I20260810 01:47:21.680517 21916 consensus_queue.cc:260] T 0f1a8e78ba774996be13eb46a950cb12 P b89e2268ec7242c992c04fc4f68afdeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } } peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } }
I20260810 01:47:21.681351 21638 heartbeater.cc:503] Master 127.9.77.190:46517 was elected leader, sending a full tablet report...
I20260810 01:47:21.681828 21916 ts_tablet_manager.cc:1428] T 0f1a8e78ba774996be13eb46a950cb12 P b89e2268ec7242c992c04fc4f68afdeb: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 01:47:21.683240 21914 raft_consensus.cc:348] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } } peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } }
I20260810 01:47:21.683574 21914 raft_consensus.cc:374] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:21.683652 21914 raft_consensus.cc:729] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86a959ed6f54447bf5c79b8dc35f9f2, State: Initialized, Role: FOLLOWER
I20260810 01:47:21.683920 21913 raft_consensus.cc:348] T 0f1a8e78ba774996be13eb46a950cb12 P a505fe46057f4a7da2c7ce013cb68638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } } peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } }
I20260810 01:47:21.684218 21913 raft_consensus.cc:374] T 0f1a8e78ba774996be13eb46a950cb12 P a505fe46057f4a7da2c7ce013cb68638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:21.684280 21913 raft_consensus.cc:729] T 0f1a8e78ba774996be13eb46a950cb12 P a505fe46057f4a7da2c7ce013cb68638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505fe46057f4a7da2c7ce013cb68638, State: Initialized, Role: FOLLOWER
I20260810 01:47:21.684221 21914 consensus_queue.cc:260] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } } peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } }
I20260810 01:47:21.685249 21913 consensus_queue.cc:260] T 0f1a8e78ba774996be13eb46a950cb12 P a505fe46057f4a7da2c7ce013cb68638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } } peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } }
I20260810 01:47:21.685452 21914 ts_tablet_manager.cc:1428] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260810 01:47:21.685598 21866 heartbeater.cc:503] Master 127.9.77.190:46517 was elected leader, sending a full tablet report...
I20260810 01:47:21.686128 21913 ts_tablet_manager.cc:1428] T 0f1a8e78ba774996be13eb46a950cb12 P a505fe46057f4a7da2c7ce013cb68638: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 01:47:21.686486 21420 heartbeater.cc:503] Master 127.9.77.190:46517 was elected leader, sending a full tablet report...
W20260810 01:47:21.787812 21421 tablet.cc:2369] T 0f1a8e78ba774996be13eb46a950cb12 P a505fe46057f4a7da2c7ce013cb68638: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 01:47:21.804862 21641 tablet.cc:2369] T 0f1a8e78ba774996be13eb46a950cb12 P b89e2268ec7242c992c04fc4f68afdeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 01:47:21.822495 21868 tablet.cc:2369] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 01:47:21.872416 21922 raft_consensus.cc:482] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:47:21.872793 21922 raft_consensus.cc:504] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } } peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } }
I20260810 01:47:21.874521 21922 leader_election.cc:288] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b89e2268ec7242c992c04fc4f68afdeb (127.9.77.130:44931), a505fe46057f4a7da2c7ce013cb68638 (127.9.77.129:41641)
I20260810 01:47:21.879077 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:21.874978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:44931 (local address 127.9.77.131:42339)
0810 01:47:21.875185 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:21.875206 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:21.875357 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:21.875728 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:21.875731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:21.875748 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:21.875937 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:21.875945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.876854 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.876860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:21.877861 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:21.877877 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.878580 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.878584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:21.878691 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:21.878717 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 01:47:21.878817 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 01:47:21.878915 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 01:47:21.879077 21906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:21.875048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:42339 (local address 127.9.77.130:44931)
0810 01:47:21.875234 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:21.875241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:21.875271 (+ 30us) server_negotiation.cc:407] Connection header received
0810 01:47:21.875521 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:21.875528 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:21.875604 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:21.875702 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:21.876086 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.876736 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.877996 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.878440 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.878580 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:21.878650 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:21.878834 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 01:47:21.878912 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 01:47:21.879799 21926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:21.875332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:41641 (local address 127.9.77.131:55063)
0810 01:47:21.875803 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:21.875824 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:21.875948 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:21.876264 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:21.876266 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:21.876282 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:21.876841 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:21.876850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.877675 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.877680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:21.878490 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:21.878499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.879396 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.879400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:21.879495 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:21.879520 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 01:47:21.879607 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 01:47:21.879678 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":186,"threads_started":1}
I20260810 01:47:21.879848 21910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:21.875585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:55063 (local address 127.9.77.129:41641)
0810 01:47:21.875762 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:21.875767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:21.875910 (+ 143us) server_negotiation.cc:407] Connection header received
0810 01:47:21.876093 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:21.876096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:21.876158 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:21.876251 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:21.876976 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.877536 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.878622 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.879096 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.879250 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:21.879322 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:21.879631 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 01:47:21.879719 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 01:47:21.879932 21564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1a8e78ba774996be13eb46a950cb12" candidate_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89e2268ec7242c992c04fc4f68afdeb" is_pre_election: true
I20260810 01:47:21.880403 21564 raft_consensus.cc:2454] T 0f1a8e78ba774996be13eb46a950cb12 P b89e2268ec7242c992c04fc4f68afdeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86a959ed6f54447bf5c79b8dc35f9f2 in term 0.
I20260810 01:47:21.880689 21343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1a8e78ba774996be13eb46a950cb12" candidate_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505fe46057f4a7da2c7ce013cb68638" is_pre_election: true
I20260810 01:47:21.881050 21343 raft_consensus.cc:2454] T 0f1a8e78ba774996be13eb46a950cb12 P a505fe46057f4a7da2c7ce013cb68638 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86a959ed6f54447bf5c79b8dc35f9f2 in term 0.
I20260810 01:47:21.881047 21679 leader_election.cc:302] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b89e2268ec7242c992c04fc4f68afdeb, c86a959ed6f54447bf5c79b8dc35f9f2; no voters:
I20260810 01:47:21.881407 21922 raft_consensus.cc:2781] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:47:21.881529 21922 raft_consensus.cc:482] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:47:21.881567 21922 raft_consensus.cc:3037] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:21.882939 21922 raft_consensus.cc:504] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } } peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } }
I20260810 01:47:21.883337 21922 leader_election.cc:288] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2 [CANDIDATE]: Term 1 election: Requested vote from peers b89e2268ec7242c992c04fc4f68afdeb (127.9.77.130:44931), a505fe46057f4a7da2c7ce013cb68638 (127.9.77.129:41641)
I20260810 01:47:21.883756 21343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1a8e78ba774996be13eb46a950cb12" candidate_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505fe46057f4a7da2c7ce013cb68638"
I20260810 01:47:21.883736 21564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1a8e78ba774996be13eb46a950cb12" candidate_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89e2268ec7242c992c04fc4f68afdeb"
I20260810 01:47:21.883890 21564 raft_consensus.cc:3037] T 0f1a8e78ba774996be13eb46a950cb12 P b89e2268ec7242c992c04fc4f68afdeb [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:21.883890 21343 raft_consensus.cc:3037] T 0f1a8e78ba774996be13eb46a950cb12 P a505fe46057f4a7da2c7ce013cb68638 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:21.885440 21343 raft_consensus.cc:2454] T 0f1a8e78ba774996be13eb46a950cb12 P a505fe46057f4a7da2c7ce013cb68638 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86a959ed6f54447bf5c79b8dc35f9f2 in term 1.
I20260810 01:47:21.885449 21564 raft_consensus.cc:2454] T 0f1a8e78ba774996be13eb46a950cb12 P b89e2268ec7242c992c04fc4f68afdeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86a959ed6f54447bf5c79b8dc35f9f2 in term 1.
I20260810 01:47:21.885783 21679 leader_election.cc:302] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a505fe46057f4a7da2c7ce013cb68638, c86a959ed6f54447bf5c79b8dc35f9f2; no voters:
I20260810 01:47:21.886019 21922 raft_consensus.cc:2781] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:47:21.886320 21922 raft_consensus.cc:686] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 1 LEADER]: Becoming Leader. State: Replica: c86a959ed6f54447bf5c79b8dc35f9f2, State: Running, Role: LEADER
I20260810 01:47:21.886755 21922 consensus_queue.cc:237] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } } peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } }
I20260810 01:47:21.890781 21092 catalog_manager.cc:5495] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86a959ed6f54447bf5c79b8dc35f9f2 (127.9.77.131). New cstate: current_term: 1 leader_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } health_report { overall_health: UNKNOWN } } }
I20260810 01:47:21.910398 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:21.906341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60684 (local address 127.9.77.131:35369)
0810 01:47:21.906514 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:21.906518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:21.906535 (+ 17us) server_negotiation.cc:407] Connection header received
0810 01:47:21.906633 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:21.906636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:21.906704 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:21.906807 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:47:21.907108 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.907745 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.908742 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:21.908974 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:21.909064 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:21.909315 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:47:21.909937 (+ 622us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:47:21.910036 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:21.910091 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 01:47:21.910155 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 01:47:21.918470 21343 raft_consensus.cc:1261] T 0f1a8e78ba774996be13eb46a950cb12 P a505fe46057f4a7da2c7ce013cb68638 [term 1 FOLLOWER]: Refusing update from remote peer c86a959ed6f54447bf5c79b8dc35f9f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:47:21.918470 21564 raft_consensus.cc:1261] T 0f1a8e78ba774996be13eb46a950cb12 P b89e2268ec7242c992c04fc4f68afdeb [term 1 FOLLOWER]: Refusing update from remote peer c86a959ed6f54447bf5c79b8dc35f9f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:47:21.919380 21929 consensus_queue.cc:1035] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:47:21.919674 21922 consensus_queue.cc:1035] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:47:21.930027 21935 mvcc.cc:205] Tried to move back new op lower bound from 7316793106087780352 to 7316793105971523584. Current Snapshot: MvccSnapshot[applied={T|T < 7316793106087780352}]
I20260810 01:47:21.941946 21090 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:47:21.942188 21090 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:47:21.945432 21090 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fab89cac2fe444da5e6f375fef7e249: Sending DeleteTablet for 3 replicas of tablet 0f1a8e78ba774996be13eb46a950cb12
I20260810 01:47:21.946488 21751 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f1a8e78ba774996be13eb46a950cb12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:47:21 UTC) from {username='oryx', principal='oryx/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:60672
I20260810 01:47:21.946542 21534 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f1a8e78ba774996be13eb46a950cb12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:47:21 UTC) from {username='oryx', principal='oryx/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:60000
I20260810 01:47:21.947269 21090 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 01:47:21.947333 21313 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f1a8e78ba774996be13eb46a950cb12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:47:21 UTC) from {username='oryx', principal='oryx/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:50576
W20260810 01:47:21.947829 21090 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:47:21.947934 21954 tablet_replica.cc:330] T 0f1a8e78ba774996be13eb46a950cb12 P b89e2268ec7242c992c04fc4f68afdeb: stopping tablet replica
I20260810 01:47:21.948433 21954 raft_consensus.cc:2229] T 0f1a8e78ba774996be13eb46a950cb12 P b89e2268ec7242c992c04fc4f68afdeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:47:21.948873 21954 raft_consensus.cc:2258] T 0f1a8e78ba774996be13eb46a950cb12 P b89e2268ec7242c992c04fc4f68afdeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:47:21.949340 21957 tablet_replica.cc:330] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2: stopping tablet replica
I20260810 01:47:21.949669 21957 raft_consensus.cc:2229] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 1 LEADER]: Raft consensus shutting down.
I20260810 01:47:21.950117 21957 raft_consensus.cc:2258] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:47:21.951975 21957 ts_tablet_manager.cc:1905] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:47:21.953392 21534 tablet_service.cc:1467] Processing CreateTablet for tablet 915d7e917035477b8cd38dcb728540fa (DEFAULT_TABLE table=test-table [id=7b702419191f4934bcb6a392a1e51065]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:47:21.953492 21751 tablet_service.cc:1467] Processing CreateTablet for tablet 915d7e917035477b8cd38dcb728540fa (DEFAULT_TABLE table=test-table [id=7b702419191f4934bcb6a392a1e51065]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:47:21.953747 21751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915d7e917035477b8cd38dcb728540fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:21.953747 21534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915d7e917035477b8cd38dcb728540fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:21.955183 21957 ts_tablet_manager.cc:1918] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:47:21.955291 21957 log.cc:1196] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-2/wal/wals/0f1a8e78ba774996be13eb46a950cb12
I20260810 01:47:21.955629 21957 ts_tablet_manager.cc:1939] T 0f1a8e78ba774996be13eb46a950cb12 P c86a959ed6f54447bf5c79b8dc35f9f2: Deleting consensus metadata
I20260810 01:47:21.956367 21914 tablet_bootstrap.cc:492] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2: Bootstrap starting.
I20260810 01:47:21.957095 21075 catalog_manager.cc:4841] TS c86a959ed6f54447bf5c79b8dc35f9f2 (127.9.77.131:35369): tablet 0f1a8e78ba774996be13eb46a950cb12 (table test-table [id=a84b873b6c244723a7ca0cc24e6070b5]) successfully deleted
I20260810 01:47:21.957433 21914 tablet_bootstrap.cc:654] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:21.957639 21313 tablet_service.cc:1467] Processing CreateTablet for tablet 915d7e917035477b8cd38dcb728540fa (DEFAULT_TABLE table=test-table [id=7b702419191f4934bcb6a392a1e51065]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:47:21.957880 21916 tablet_bootstrap.cc:492] T 915d7e917035477b8cd38dcb728540fa P b89e2268ec7242c992c04fc4f68afdeb: Bootstrap starting.
I20260810 01:47:21.958086 21313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915d7e917035477b8cd38dcb728540fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:21.958212 21954 ts_tablet_manager.cc:1905] T 0f1a8e78ba774996be13eb46a950cb12 P b89e2268ec7242c992c04fc4f68afdeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:47:21.958915 21916 tablet_bootstrap.cc:654] T 915d7e917035477b8cd38dcb728540fa P b89e2268ec7242c992c04fc4f68afdeb: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:21.960789 21916 tablet_bootstrap.cc:492] T 915d7e917035477b8cd38dcb728540fa P b89e2268ec7242c992c04fc4f68afdeb: No bootstrap required, opened a new log
I20260810 01:47:21.960889 21916 ts_tablet_manager.cc:1397] T 915d7e917035477b8cd38dcb728540fa P b89e2268ec7242c992c04fc4f68afdeb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260810 01:47:21.961215 21913 tablet_bootstrap.cc:492] T 915d7e917035477b8cd38dcb728540fa P a505fe46057f4a7da2c7ce013cb68638: Bootstrap starting.
I20260810 01:47:21.961401 21916 raft_consensus.cc:348] T 915d7e917035477b8cd38dcb728540fa P b89e2268ec7242c992c04fc4f68afdeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:21.961544 21916 raft_consensus.cc:374] T 915d7e917035477b8cd38dcb728540fa P b89e2268ec7242c992c04fc4f68afdeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:21.961578 21916 raft_consensus.cc:729] T 915d7e917035477b8cd38dcb728540fa P b89e2268ec7242c992c04fc4f68afdeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89e2268ec7242c992c04fc4f68afdeb, State: Initialized, Role: FOLLOWER
I20260810 01:47:21.961733 21916 consensus_queue.cc:260] T 915d7e917035477b8cd38dcb728540fa P b89e2268ec7242c992c04fc4f68afdeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:21.962038 21916 ts_tablet_manager.cc:1428] T 915d7e917035477b8cd38dcb728540fa P b89e2268ec7242c992c04fc4f68afdeb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 01:47:21.962291 21913 tablet_bootstrap.cc:654] T 915d7e917035477b8cd38dcb728540fa P a505fe46057f4a7da2c7ce013cb68638: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:21.956748 21960 tablet_replica.cc:330] T 0f1a8e78ba774996be13eb46a950cb12 P a505fe46057f4a7da2c7ce013cb68638: stopping tablet replica
I20260810 01:47:21.962972 21960 raft_consensus.cc:2229] T 0f1a8e78ba774996be13eb46a950cb12 P a505fe46057f4a7da2c7ce013cb68638 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:47:21.963124 21954 ts_tablet_manager.cc:1918] T 0f1a8e78ba774996be13eb46a950cb12 P b89e2268ec7242c992c04fc4f68afdeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:47:21.963222 21954 log.cc:1196] T 0f1a8e78ba774996be13eb46a950cb12 P b89e2268ec7242c992c04fc4f68afdeb: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-1/wal/wals/0f1a8e78ba774996be13eb46a950cb12
I20260810 01:47:21.963394 21960 raft_consensus.cc:2258] T 0f1a8e78ba774996be13eb46a950cb12 P a505fe46057f4a7da2c7ce013cb68638 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:47:21.963568 21954 ts_tablet_manager.cc:1939] T 0f1a8e78ba774996be13eb46a950cb12 P b89e2268ec7242c992c04fc4f68afdeb: Deleting consensus metadata
I20260810 01:47:21.965068 21073 catalog_manager.cc:4841] TS b89e2268ec7242c992c04fc4f68afdeb (127.9.77.130:44931): tablet 0f1a8e78ba774996be13eb46a950cb12 (table test-table [id=a84b873b6c244723a7ca0cc24e6070b5]) successfully deleted
I20260810 01:47:21.965766 21960 ts_tablet_manager.cc:1905] T 0f1a8e78ba774996be13eb46a950cb12 P a505fe46057f4a7da2c7ce013cb68638: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:47:21.966090 21913 tablet_bootstrap.cc:492] T 915d7e917035477b8cd38dcb728540fa P a505fe46057f4a7da2c7ce013cb68638: No bootstrap required, opened a new log
I20260810 01:47:21.966187 21913 ts_tablet_manager.cc:1397] T 915d7e917035477b8cd38dcb728540fa P a505fe46057f4a7da2c7ce013cb68638: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260810 01:47:21.966711 21913 raft_consensus.cc:348] T 915d7e917035477b8cd38dcb728540fa P a505fe46057f4a7da2c7ce013cb68638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:21.966840 21913 raft_consensus.cc:374] T 915d7e917035477b8cd38dcb728540fa P a505fe46057f4a7da2c7ce013cb68638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:21.966882 21913 raft_consensus.cc:729] T 915d7e917035477b8cd38dcb728540fa P a505fe46057f4a7da2c7ce013cb68638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505fe46057f4a7da2c7ce013cb68638, State: Initialized, Role: FOLLOWER
I20260810 01:47:21.967029 21913 consensus_queue.cc:260] T 915d7e917035477b8cd38dcb728540fa P a505fe46057f4a7da2c7ce013cb68638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:21.967361 21913 ts_tablet_manager.cc:1428] T 915d7e917035477b8cd38dcb728540fa P a505fe46057f4a7da2c7ce013cb68638: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 01:47:21.969413 21960 ts_tablet_manager.cc:1918] T 0f1a8e78ba774996be13eb46a950cb12 P a505fe46057f4a7da2c7ce013cb68638: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:47:21.969529 21960 log.cc:1196] T 0f1a8e78ba774996be13eb46a950cb12 P a505fe46057f4a7da2c7ce013cb68638: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-0/wal/wals/0f1a8e78ba774996be13eb46a950cb12
I20260810 01:47:21.969899 21960 ts_tablet_manager.cc:1939] T 0f1a8e78ba774996be13eb46a950cb12 P a505fe46057f4a7da2c7ce013cb68638: Deleting consensus metadata
I20260810 01:47:21.971202 21073 catalog_manager.cc:4841] TS a505fe46057f4a7da2c7ce013cb68638 (127.9.77.129:41641): tablet 0f1a8e78ba774996be13eb46a950cb12 (table test-table [id=a84b873b6c244723a7ca0cc24e6070b5]) successfully deleted
I20260810 01:47:21.972134 21914 tablet_bootstrap.cc:492] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2: No bootstrap required, opened a new log
I20260810 01:47:21.972250 21914 ts_tablet_manager.cc:1397] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.001s
I20260810 01:47:21.972818 21914 raft_consensus.cc:348] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:21.972961 21914 raft_consensus.cc:374] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:21.972993 21914 raft_consensus.cc:729] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86a959ed6f54447bf5c79b8dc35f9f2, State: Initialized, Role: FOLLOWER
I20260810 01:47:21.973155 21914 consensus_queue.cc:260] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:21.973459 21914 ts_tablet_manager.cc:1428] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 01:47:22.063789 21929 raft_consensus.cc:482] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:47:22.063999 21929 raft_consensus.cc:504] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.064682 21929 leader_election.cc:288] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a505fe46057f4a7da2c7ce013cb68638 (127.9.77.129:41641), b89e2268ec7242c992c04fc4f68afdeb (127.9.77.130:44931)
I20260810 01:47:22.065281 21343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915d7e917035477b8cd38dcb728540fa" candidate_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505fe46057f4a7da2c7ce013cb68638" is_pre_election: true
I20260810 01:47:22.065371 21564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915d7e917035477b8cd38dcb728540fa" candidate_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89e2268ec7242c992c04fc4f68afdeb" is_pre_election: true
I20260810 01:47:22.065495 21343 raft_consensus.cc:2454] T 915d7e917035477b8cd38dcb728540fa P a505fe46057f4a7da2c7ce013cb68638 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86a959ed6f54447bf5c79b8dc35f9f2 in term 0.
I20260810 01:47:22.065608 21564 raft_consensus.cc:2454] T 915d7e917035477b8cd38dcb728540fa P b89e2268ec7242c992c04fc4f68afdeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86a959ed6f54447bf5c79b8dc35f9f2 in term 0.
I20260810 01:47:22.065902 21679 leader_election.cc:302] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a505fe46057f4a7da2c7ce013cb68638, c86a959ed6f54447bf5c79b8dc35f9f2; no voters:
I20260810 01:47:22.066112 21929 raft_consensus.cc:2781] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:47:22.066207 21929 raft_consensus.cc:482] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:47:22.066260 21929 raft_consensus.cc:3037] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:22.067605 21929 raft_consensus.cc:504] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.067979 21929 leader_election.cc:288] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2 [CANDIDATE]: Term 1 election: Requested vote from peers a505fe46057f4a7da2c7ce013cb68638 (127.9.77.129:41641), b89e2268ec7242c992c04fc4f68afdeb (127.9.77.130:44931)
I20260810 01:47:22.068487 21564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915d7e917035477b8cd38dcb728540fa" candidate_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89e2268ec7242c992c04fc4f68afdeb"
I20260810 01:47:22.068491 21343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "915d7e917035477b8cd38dcb728540fa" candidate_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505fe46057f4a7da2c7ce013cb68638"
I20260810 01:47:22.068652 21343 raft_consensus.cc:3037] T 915d7e917035477b8cd38dcb728540fa P a505fe46057f4a7da2c7ce013cb68638 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:22.068656 21564 raft_consensus.cc:3037] T 915d7e917035477b8cd38dcb728540fa P b89e2268ec7242c992c04fc4f68afdeb [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:22.069891 21564 raft_consensus.cc:2454] T 915d7e917035477b8cd38dcb728540fa P b89e2268ec7242c992c04fc4f68afdeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86a959ed6f54447bf5c79b8dc35f9f2 in term 1.
I20260810 01:47:22.070132 21343 raft_consensus.cc:2454] T 915d7e917035477b8cd38dcb728540fa P a505fe46057f4a7da2c7ce013cb68638 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86a959ed6f54447bf5c79b8dc35f9f2 in term 1.
I20260810 01:47:22.070248 21679 leader_election.cc:302] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b89e2268ec7242c992c04fc4f68afdeb, c86a959ed6f54447bf5c79b8dc35f9f2; no voters:
I20260810 01:47:22.070664 21929 raft_consensus.cc:2781] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:47:22.070770 21929 raft_consensus.cc:686] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2 [term 1 LEADER]: Becoming Leader. State: Replica: c86a959ed6f54447bf5c79b8dc35f9f2, State: Running, Role: LEADER
I20260810 01:47:22.070946 21929 consensus_queue.cc:237] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.073237 21094 catalog_manager.cc:5495] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86a959ed6f54447bf5c79b8dc35f9f2 (127.9.77.131). New cstate: current_term: 1 leader_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } health_report { overall_health: HEALTHY } } }
Aug 10 01:47:22 dist-test-slave-kvfs krb5kdc[21019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326440, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.190@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM
I20260810 01:47:22.117869 21170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:22.107151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60352 (local address 127.9.77.190:46517)
0810 01:47:22.107336 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:22.107343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:22.107718 (+ 375us) server_negotiation.cc:407] Connection header received
0810 01:47:22.107846 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:22.107849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:22.107908 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:22.107986 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:22.109149 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:22.109745 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:22.110804 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:22.111047 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:22.111129 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:22.114304 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:22.114308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:22.114310 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:22.114333 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:22.116014 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:22.116639 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:22.116641 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:22.116643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:22.116726 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:22.116984 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:22.116986 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:22.116987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:22.117187 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:22.117265 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:22.117578 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 01:47:22.117730 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":61}
I20260810 01:47:22.117871 21979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:22.107077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46517 (local address 127.0.0.1:60352)
0810 01:47:22.107566 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:22.107584 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:22.107743 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:22.108053 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:22.108058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:22.108855 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:22.109005 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:22.109016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:22.109867 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:22.109871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:22.110676 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:22.110687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:22.111168 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:22.111172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:22.111252 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:22.112161 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:22.112177 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:22.114139 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:22.116185 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:22.116196 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:22.116203 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:22.116543 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:22.116808 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:22.116811 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:22.116812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:22.116905 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:22.117279 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:22.117286 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:22.117421 (+ 135us) client_negotiation.cc:769] Sending connection context
0810 01:47:22.117521 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 01:47:22.117670 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":217,"threads_started":1}
I20260810 01:47:22.122920 21094 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:60352:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 01:47:22.123993 21094 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:47:22.129976 21313 tablet_service.cc:1467] Processing CreateTablet for tablet a60e03643ec54700b74ce4124de9bb84 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ecfaf75881704b48a30e095f55b24a9b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 01:47:22.130110 21534 tablet_service.cc:1467] Processing CreateTablet for tablet a60e03643ec54700b74ce4124de9bb84 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ecfaf75881704b48a30e095f55b24a9b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 01:47:22.130550 21313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a60e03643ec54700b74ce4124de9bb84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:22.130623 21534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a60e03643ec54700b74ce4124de9bb84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:22.134111 21916 tablet_bootstrap.cc:492] T a60e03643ec54700b74ce4124de9bb84 P b89e2268ec7242c992c04fc4f68afdeb: Bootstrap starting.
I20260810 01:47:22.134339 21913 tablet_bootstrap.cc:492] T a60e03643ec54700b74ce4124de9bb84 P a505fe46057f4a7da2c7ce013cb68638: Bootstrap starting.
I20260810 01:47:22.134675 21751 tablet_service.cc:1467] Processing CreateTablet for tablet a60e03643ec54700b74ce4124de9bb84 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ecfaf75881704b48a30e095f55b24a9b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 01:47:22.135198 21751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a60e03643ec54700b74ce4124de9bb84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:22.135414 21913 tablet_bootstrap.cc:654] T a60e03643ec54700b74ce4124de9bb84 P a505fe46057f4a7da2c7ce013cb68638: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:22.135545 21916 tablet_bootstrap.cc:654] T a60e03643ec54700b74ce4124de9bb84 P b89e2268ec7242c992c04fc4f68afdeb: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:22.137566 21916 tablet_bootstrap.cc:492] T a60e03643ec54700b74ce4124de9bb84 P b89e2268ec7242c992c04fc4f68afdeb: No bootstrap required, opened a new log
I20260810 01:47:22.137651 21916 ts_tablet_manager.cc:1397] T a60e03643ec54700b74ce4124de9bb84 P b89e2268ec7242c992c04fc4f68afdeb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 01:47:22.138077 21916 raft_consensus.cc:348] T a60e03643ec54700b74ce4124de9bb84 P b89e2268ec7242c992c04fc4f68afdeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.138190 21916 raft_consensus.cc:374] T a60e03643ec54700b74ce4124de9bb84 P b89e2268ec7242c992c04fc4f68afdeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:22.138213 21916 raft_consensus.cc:729] T a60e03643ec54700b74ce4124de9bb84 P b89e2268ec7242c992c04fc4f68afdeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89e2268ec7242c992c04fc4f68afdeb, State: Initialized, Role: FOLLOWER
I20260810 01:47:22.138367 21916 consensus_queue.cc:260] T a60e03643ec54700b74ce4124de9bb84 P b89e2268ec7242c992c04fc4f68afdeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.138659 21921 tablet_replica.cc:439] T a60e03643ec54700b74ce4124de9bb84 P b89e2268ec7242c992c04fc4f68afdeb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } } }
I20260810 01:47:22.138828 21921 tablet_replica.cc:442] T a60e03643ec54700b74ce4124de9bb84 P b89e2268ec7242c992c04fc4f68afdeb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 01:47:22.139230 21916 ts_tablet_manager.cc:1428] T a60e03643ec54700b74ce4124de9bb84 P b89e2268ec7242c992c04fc4f68afdeb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 01:47:22.139961 21914 tablet_bootstrap.cc:492] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2: Bootstrap starting.
I20260810 01:47:22.141024 21913 tablet_bootstrap.cc:492] T a60e03643ec54700b74ce4124de9bb84 P a505fe46057f4a7da2c7ce013cb68638: No bootstrap required, opened a new log
I20260810 01:47:22.141157 21913 ts_tablet_manager.cc:1397] T a60e03643ec54700b74ce4124de9bb84 P a505fe46057f4a7da2c7ce013cb68638: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260810 01:47:22.141206 21914 tablet_bootstrap.cc:654] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:22.141786 21913 raft_consensus.cc:348] T a60e03643ec54700b74ce4124de9bb84 P a505fe46057f4a7da2c7ce013cb68638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.141961 21913 raft_consensus.cc:374] T a60e03643ec54700b74ce4124de9bb84 P a505fe46057f4a7da2c7ce013cb68638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:22.142015 21913 raft_consensus.cc:729] T a60e03643ec54700b74ce4124de9bb84 P a505fe46057f4a7da2c7ce013cb68638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505fe46057f4a7da2c7ce013cb68638, State: Initialized, Role: FOLLOWER
I20260810 01:47:22.142226 21913 consensus_queue.cc:260] T a60e03643ec54700b74ce4124de9bb84 P a505fe46057f4a7da2c7ce013cb68638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.142521 21914 tablet_bootstrap.cc:492] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2: No bootstrap required, opened a new log
I20260810 01:47:22.142628 21914 ts_tablet_manager.cc:1397] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 01:47:22.142542 21924 tablet_replica.cc:439] T a60e03643ec54700b74ce4124de9bb84 P a505fe46057f4a7da2c7ce013cb68638: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } } }
I20260810 01:47:22.142719 21924 tablet_replica.cc:442] T a60e03643ec54700b74ce4124de9bb84 P a505fe46057f4a7da2c7ce013cb68638: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 01:47:22.143092 21914 raft_consensus.cc:348] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.143225 21914 raft_consensus.cc:374] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:22.143263 21914 raft_consensus.cc:729] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86a959ed6f54447bf5c79b8dc35f9f2, State: Initialized, Role: FOLLOWER
I20260810 01:47:22.143276 21913 ts_tablet_manager.cc:1428] T a60e03643ec54700b74ce4124de9bb84 P a505fe46057f4a7da2c7ce013cb68638: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 01:47:22.143400 21914 consensus_queue.cc:260] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.143708 21914 ts_tablet_manager.cc:1428] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 01:47:22.143934 21929 tablet_replica.cc:439] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } } }
I20260810 01:47:22.144129 21929 tablet_replica.cc:442] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 01:47:22.422266 21924 raft_consensus.cc:482] T a60e03643ec54700b74ce4124de9bb84 P a505fe46057f4a7da2c7ce013cb68638 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:47:22.422557 21924 raft_consensus.cc:504] T a60e03643ec54700b74ce4124de9bb84 P a505fe46057f4a7da2c7ce013cb68638 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.424185 21924 leader_election.cc:288] T a60e03643ec54700b74ce4124de9bb84 P a505fe46057f4a7da2c7ce013cb68638 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b89e2268ec7242c992c04fc4f68afdeb (127.9.77.130:44931), c86a959ed6f54447bf5c79b8dc35f9f2 (127.9.77.131:35369)
I20260810 01:47:22.425503 21929 raft_consensus.cc:482] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:47:22.425648 21929 raft_consensus.cc:504] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.426149 21929 leader_election.cc:288] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a505fe46057f4a7da2c7ce013cb68638 (127.9.77.129:41641), b89e2268ec7242c992c04fc4f68afdeb (127.9.77.130:44931)
I20260810 01:47:22.426620 21343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60e03643ec54700b74ce4124de9bb84" candidate_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505fe46057f4a7da2c7ce013cb68638" is_pre_election: true
I20260810 01:47:22.426699 21564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60e03643ec54700b74ce4124de9bb84" candidate_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89e2268ec7242c992c04fc4f68afdeb" is_pre_election: true
I20260810 01:47:22.426780 21343 raft_consensus.cc:2454] T a60e03643ec54700b74ce4124de9bb84 P a505fe46057f4a7da2c7ce013cb68638 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86a959ed6f54447bf5c79b8dc35f9f2 in term 0.
I20260810 01:47:22.426903 21564 raft_consensus.cc:2454] T a60e03643ec54700b74ce4124de9bb84 P b89e2268ec7242c992c04fc4f68afdeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86a959ed6f54447bf5c79b8dc35f9f2 in term 0.
I20260810 01:47:22.427130 21679 leader_election.cc:302] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a505fe46057f4a7da2c7ce013cb68638, c86a959ed6f54447bf5c79b8dc35f9f2; no voters:
I20260810 01:47:22.427472 21929 raft_consensus.cc:2781] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:47:22.427539 21929 raft_consensus.cc:482] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:47:22.427582 21929 raft_consensus.cc:3037] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:22.429247 21929 raft_consensus.cc:504] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.429643 21929 leader_election.cc:288] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2 [CANDIDATE]: Term 1 election: Requested vote from peers a505fe46057f4a7da2c7ce013cb68638 (127.9.77.129:41641), b89e2268ec7242c992c04fc4f68afdeb (127.9.77.130:44931)
I20260810 01:47:22.429742 21990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:22.424527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:44931 (local address 127.9.77.129:43331)
0810 01:47:22.425099 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:22.425112 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:22.425197 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:22.425558 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:22.425561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:22.425589 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:22.425734 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:22.425753 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:22.426923 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:22.426926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:22.428290 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:22.428301 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:22.429208 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:22.429212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:22.429309 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:22.429335 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 01:47:22.429443 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 01:47:22.429542 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":143,"thread_start_us":121,"threads_started":1}
I20260810 01:47:22.429993 21992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:22.424770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:43331 (local address 127.9.77.130:44931)
0810 01:47:22.425109 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:22.425115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:22.425172 (+ 57us) server_negotiation.cc:407] Connection header received
0810 01:47:22.425321 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:22.425325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:22.425410 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:22.425516 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:22.425931 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:22.426780 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:22.428431 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:22.429068 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:22.429208 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:22.429278 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:22.429506 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 01:47:22.429604 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":114,"threads_started":1}
I20260810 01:47:22.430157 21564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60e03643ec54700b74ce4124de9bb84" candidate_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89e2268ec7242c992c04fc4f68afdeb"
I20260810 01:47:22.430159 21343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60e03643ec54700b74ce4124de9bb84" candidate_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505fe46057f4a7da2c7ce013cb68638"
I20260810 01:47:22.430292 21564 raft_consensus.cc:3037] T a60e03643ec54700b74ce4124de9bb84 P b89e2268ec7242c992c04fc4f68afdeb [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:22.430307 21343 raft_consensus.cc:3037] T a60e03643ec54700b74ce4124de9bb84 P a505fe46057f4a7da2c7ce013cb68638 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:22.430454 21563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60e03643ec54700b74ce4124de9bb84" candidate_uuid: "a505fe46057f4a7da2c7ce013cb68638" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89e2268ec7242c992c04fc4f68afdeb" is_pre_election: true
I20260810 01:47:22.430945 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:22.424527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:35369 (local address 127.9.77.129:44151)
0810 01:47:22.424985 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:22.425007 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:22.425158 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:22.425677 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:22.425683 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:22.425705 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:22.425847 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:22.425854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:22.427496 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:22.427499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:22.428719 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:22.428730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:22.430310 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:22.430315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:22.430620 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:22.430648 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 01:47:22.430721 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 01:47:22.430806 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":100,"threads_started":1}
I20260810 01:47:22.431316 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:22.424797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:44151 (local address 127.9.77.131:35369)
0810 01:47:22.425109 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:22.425115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:22.425140 (+ 25us) server_negotiation.cc:407] Connection header received
0810 01:47:22.425321 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:22.425325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:22.425392 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:22.425516 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:22.426313 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:22.427302 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:22.428859 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:22.430062 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:22.430186 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:22.430245 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:22.431095 (+ 850us) server_negotiation.cc:299] Negotiation successful
0810 01:47:22.431189 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":95,"threads_started":1}
I20260810 01:47:22.431751 21343 raft_consensus.cc:2454] T a60e03643ec54700b74ce4124de9bb84 P a505fe46057f4a7da2c7ce013cb68638 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86a959ed6f54447bf5c79b8dc35f9f2 in term 1.
I20260810 01:47:22.431775 21564 raft_consensus.cc:2454] T a60e03643ec54700b74ce4124de9bb84 P b89e2268ec7242c992c04fc4f68afdeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86a959ed6f54447bf5c79b8dc35f9f2 in term 1.
I20260810 01:47:22.432081 21679 leader_election.cc:302] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a505fe46057f4a7da2c7ce013cb68638, c86a959ed6f54447bf5c79b8dc35f9f2; no voters:
I20260810 01:47:22.432085 21784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a60e03643ec54700b74ce4124de9bb84" candidate_uuid: "a505fe46057f4a7da2c7ce013cb68638" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" is_pre_election: true
I20260810 01:47:22.432410 21784 raft_consensus.cc:2379] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a505fe46057f4a7da2c7ce013cb68638 in current term 1: Already voted for candidate c86a959ed6f54447bf5c79b8dc35f9f2 in this term.
I20260810 01:47:22.432647 21929 raft_consensus.cc:2781] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:47:22.432773 21929 raft_consensus.cc:686] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 1 LEADER]: Becoming Leader. State: Replica: c86a959ed6f54447bf5c79b8dc35f9f2, State: Running, Role: LEADER
I20260810 01:47:22.433015 21239 leader_election.cc:302] T a60e03643ec54700b74ce4124de9bb84 P a505fe46057f4a7da2c7ce013cb68638 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a505fe46057f4a7da2c7ce013cb68638; no voters: b89e2268ec7242c992c04fc4f68afdeb, c86a959ed6f54447bf5c79b8dc35f9f2
I20260810 01:47:22.433202 21929 consensus_queue.cc:237] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.433307 21924 raft_consensus.cc:2726] T a60e03643ec54700b74ce4124de9bb84 P a505fe46057f4a7da2c7ce013cb68638 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 01:47:22.434269 21922 tablet_replica.cc:439] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2: TxnStatusTablet state changed. Reason: New leader c86a959ed6f54447bf5c79b8dc35f9f2. Latest consensus state: current_term: 1 leader_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } } }
I20260810 01:47:22.434394 21922 tablet_replica.cc:442] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2: This TxnStatusTablet replica's current role is: LEADER
I20260810 01:47:22.434652 21995 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 01:47:22.434840 21995 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 01:47:22.435621 21090 catalog_manager.cc:5495] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86a959ed6f54447bf5c79b8dc35f9f2 (127.9.77.131). New cstate: current_term: 1 leader_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } health_report { overall_health: HEALTHY } } }
I20260810 01:47:22.499015 21922 consensus_queue.cc:1035] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:47:22.504151 21929 consensus_queue.cc:1035] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:47:22.511973 21979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:22.508225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:35369 (local address 127.0.0.1:60698)
0810 01:47:22.508387 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:22.508406 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:22.508576 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:22.508919 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:22.508922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:22.508932 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 01:47:22.509087 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:22.509092 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:22.509916 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:22.509920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:22.510607 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:22.510615 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:22.511118 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:22.511122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:22.511216 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:22.511285 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 01:47:22.511636 (+ 351us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 01:47:22.511658 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 01:47:22.511726 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 01:47:22.511832 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 01:47:22.511974 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:22.508288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60698 (local address 127.9.77.131:35369)
0810 01:47:22.508405 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:22.508412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:22.508491 (+ 79us) server_negotiation.cc:407] Connection header received
0810 01:47:22.508709 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:22.508714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:22.508782 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:22.508872 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:47:22.509196 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:22.509784 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:22.510734 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:22.511000 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:22.511115 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:22.511446 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:47:22.511545 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:47:22.511629 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:22.511739 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 01:47:22.511800 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 01:47:22.515239 21343 raft_consensus.cc:1261] T a60e03643ec54700b74ce4124de9bb84 P a505fe46057f4a7da2c7ce013cb68638 [term 1 FOLLOWER]: Refusing update from remote peer c86a959ed6f54447bf5c79b8dc35f9f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:47:22.515240 21564 raft_consensus.cc:1261] T a60e03643ec54700b74ce4124de9bb84 P b89e2268ec7242c992c04fc4f68afdeb [term 1 FOLLOWER]: Refusing update from remote peer c86a959ed6f54447bf5c79b8dc35f9f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:47:22.515715 21929 consensus_queue.cc:1035] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:47:22.515897 21922 consensus_queue.cc:1035] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:47:22.517397 21921 tablet_replica.cc:439] T a60e03643ec54700b74ce4124de9bb84 P b89e2268ec7242c992c04fc4f68afdeb: TxnStatusTablet state changed. Reason: New leader c86a959ed6f54447bf5c79b8dc35f9f2. Latest consensus state: current_term: 1 leader_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } } }
I20260810 01:47:22.517529 21921 tablet_replica.cc:442] T a60e03643ec54700b74ce4124de9bb84 P b89e2268ec7242c992c04fc4f68afdeb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 01:47:22.518445 22006 tablet_replica.cc:439] T a60e03643ec54700b74ce4124de9bb84 P a505fe46057f4a7da2c7ce013cb68638: TxnStatusTablet state changed. Reason: New leader c86a959ed6f54447bf5c79b8dc35f9f2. Latest consensus state: current_term: 1 leader_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } } }
I20260810 01:47:22.518594 22006 tablet_replica.cc:442] T a60e03643ec54700b74ce4124de9bb84 P a505fe46057f4a7da2c7ce013cb68638: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 01:47:22.521028 21922 tablet_replica.cc:439] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } } }
I20260810 01:47:22.521174 21922 tablet_replica.cc:442] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2: This TxnStatusTablet replica's current role is: LEADER
I20260810 01:47:22.521880 22006 tablet_replica.cc:439] T a60e03643ec54700b74ce4124de9bb84 P a505fe46057f4a7da2c7ce013cb68638: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } } }
I20260810 01:47:22.522017 22006 tablet_replica.cc:442] T a60e03643ec54700b74ce4124de9bb84 P a505fe46057f4a7da2c7ce013cb68638: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 01:47:22.522133 21921 tablet_replica.cc:439] T a60e03643ec54700b74ce4124de9bb84 P b89e2268ec7242c992c04fc4f68afdeb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } } }
I20260810 01:47:22.522256 21921 tablet_replica.cc:442] T a60e03643ec54700b74ce4124de9bb84 P b89e2268ec7242c992c04fc4f68afdeb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 01:47:22.522526 21929 tablet_replica.cc:439] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } } }
I20260810 01:47:22.522663 21929 tablet_replica.cc:442] T a60e03643ec54700b74ce4124de9bb84 P c86a959ed6f54447bf5c79b8dc35f9f2: This TxnStatusTablet replica's current role is: LEADER
I20260810 01:47:22.525384 22005 mvcc.cc:205] Tried to move back new op lower bound from 7316793108536827904 to 7316793108208218112. Current Snapshot: MvccSnapshot[applied={T|T < 7316793108536827904 or (T in {7316793108536827904})}]
I20260810 01:47:22.525656 22001 mvcc.cc:205] Tried to move back new op lower bound from 7316793108536827904 to 7316793108208218112. Current Snapshot: MvccSnapshot[applied={T|T < 7316793108536827904}]
I20260810 01:47:22.542820 22018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:22.538489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:35369 (local address 127.9.77.131:51709)
0810 01:47:22.538841 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:22.538859 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:22.539002 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:22.539258 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:22.539261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:22.539269 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 01:47:22.539406 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:22.539412 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:22.540507 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:22.540511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:22.541396 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:22.541408 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:22.541994 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:22.541997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:22.542077 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:22.542136 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 01:47:22.542466 (+ 330us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 01:47:22.542474 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 01:47:22.542537 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 01:47:22.542629 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":258,"threads_started":1}
I20260810 01:47:22.542820 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:22.538616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:51709 (local address 127.9.77.131:35369)
0810 01:47:22.538748 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:22.538751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:22.538953 (+ 202us) server_negotiation.cc:407] Connection header received
0810 01:47:22.539115 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:22.539118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:22.539163 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:22.539219 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:47:22.539554 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:22.540375 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:22.541535 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:22.541889 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:22.541995 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:22.542277 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:47:22.542375 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:47:22.542452 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:22.542544 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 01:47:22.542610 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":31}
I20260810 01:47:22.603399 21094 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:47:22.603613 21094 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:47:22.606161 21094 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fab89cac2fe444da5e6f375fef7e249: Sending DeleteTablet for 3 replicas of tablet 915d7e917035477b8cd38dcb728540fa
I20260810 01:47:22.606901 21313 tablet_service.cc:1514] Processing DeleteTablet for tablet 915d7e917035477b8cd38dcb728540fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:47:22 UTC) from {username='oryx', principal='oryx/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:50576
I20260810 01:47:22.606963 21751 tablet_service.cc:1514] Processing DeleteTablet for tablet 915d7e917035477b8cd38dcb728540fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:47:22 UTC) from {username='oryx', principal='oryx/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:60672
I20260810 01:47:22.606967 21534 tablet_service.cc:1514] Processing DeleteTablet for tablet 915d7e917035477b8cd38dcb728540fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:47:22 UTC) from {username='oryx', principal='oryx/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:60000
I20260810 01:47:22.607481 22028 tablet_replica.cc:330] T 915d7e917035477b8cd38dcb728540fa P b89e2268ec7242c992c04fc4f68afdeb: stopping tablet replica
I20260810 01:47:22.607679 22028 raft_consensus.cc:2229] T 915d7e917035477b8cd38dcb728540fa P b89e2268ec7242c992c04fc4f68afdeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:47:22.607740 22027 tablet_replica.cc:330] T 915d7e917035477b8cd38dcb728540fa P a505fe46057f4a7da2c7ce013cb68638: stopping tablet replica
I20260810 01:47:22.607865 22028 raft_consensus.cc:2258] T 915d7e917035477b8cd38dcb728540fa P b89e2268ec7242c992c04fc4f68afdeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:47:22.607923 22027 raft_consensus.cc:2229] T 915d7e917035477b8cd38dcb728540fa P a505fe46057f4a7da2c7ce013cb68638 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:47:22.608112 22027 raft_consensus.cc:2258] T 915d7e917035477b8cd38dcb728540fa P a505fe46057f4a7da2c7ce013cb68638 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:47:22.608408 22022 tablet_replica.cc:330] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2: stopping tablet replica
I20260810 01:47:22.608582 22022 raft_consensus.cc:2229] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2 [term 1 LEADER]: Raft consensus shutting down.
I20260810 01:47:22.608855 22022 raft_consensus.cc:2258] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:47:22.608886 22028 ts_tablet_manager.cc:1905] T 915d7e917035477b8cd38dcb728540fa P b89e2268ec7242c992c04fc4f68afdeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:47:22.609246 22027 ts_tablet_manager.cc:1905] T 915d7e917035477b8cd38dcb728540fa P a505fe46057f4a7da2c7ce013cb68638: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:47:22.612530 22022 ts_tablet_manager.cc:1905] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:47:22.612627 22028 ts_tablet_manager.cc:1918] T 915d7e917035477b8cd38dcb728540fa P b89e2268ec7242c992c04fc4f68afdeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 01:47:22.612726 22028 log.cc:1196] T 915d7e917035477b8cd38dcb728540fa P b89e2268ec7242c992c04fc4f68afdeb: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-1/wal/wals/915d7e917035477b8cd38dcb728540fa
I20260810 01:47:22.613042 22028 ts_tablet_manager.cc:1939] T 915d7e917035477b8cd38dcb728540fa P b89e2268ec7242c992c04fc4f68afdeb: Deleting consensus metadata
I20260810 01:47:22.613366 22027 ts_tablet_manager.cc:1918] T 915d7e917035477b8cd38dcb728540fa P a505fe46057f4a7da2c7ce013cb68638: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 01:47:22.613521 22027 log.cc:1196] T 915d7e917035477b8cd38dcb728540fa P a505fe46057f4a7da2c7ce013cb68638: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-0/wal/wals/915d7e917035477b8cd38dcb728540fa
I20260810 01:47:22.613873 22027 ts_tablet_manager.cc:1939] T 915d7e917035477b8cd38dcb728540fa P a505fe46057f4a7da2c7ce013cb68638: Deleting consensus metadata
I20260810 01:47:22.614006 21073 catalog_manager.cc:4841] TS b89e2268ec7242c992c04fc4f68afdeb (127.9.77.130:44931): tablet 915d7e917035477b8cd38dcb728540fa (table test-table [id=7b702419191f4934bcb6a392a1e51065]) successfully deleted
I20260810 01:47:22.614810 21073 catalog_manager.cc:4841] TS a505fe46057f4a7da2c7ce013cb68638 (127.9.77.129:41641): tablet 915d7e917035477b8cd38dcb728540fa (table test-table [id=7b702419191f4934bcb6a392a1e51065]) successfully deleted
I20260810 01:47:22.616585 22022 ts_tablet_manager.cc:1918] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 01:47:22.616683 22022 log.cc:1196] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-2/wal/wals/915d7e917035477b8cd38dcb728540fa
I20260810 01:47:22.616997 22022 ts_tablet_manager.cc:1939] T 915d7e917035477b8cd38dcb728540fa P c86a959ed6f54447bf5c79b8dc35f9f2: Deleting consensus metadata
I20260810 01:47:22.617923 21075 catalog_manager.cc:4841] TS c86a959ed6f54447bf5c79b8dc35f9f2 (127.9.77.131:35369): tablet 915d7e917035477b8cd38dcb728540fa (table test-table [id=7b702419191f4934bcb6a392a1e51065]) successfully deleted
Aug 10 01:47:22 dist-test-slave-kvfs krb5kdc[21019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 01:47:22 dist-test-slave-kvfs krb5kdc[21019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:22 dist-test-slave-kvfs krb5kdc[21019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
W20260810 01:47:22.642617 22038 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:22.635036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60360 (local address 127.9.77.190:46517)
0810 01:47:22.635372 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:22.635376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:22.635395 (+ 19us) server_negotiation.cc:407] Connection header received
0810 01:47:22.635563 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:22.635567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:22.635635 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:22.635758 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:22.636631 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:22.637317 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:22.638289 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:22.638542 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:22.638634 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:22.642452 (+ 3818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60360: BlockingRecv error: recv got EOF from 127.0.0.1:60360 (error 108)
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":96,"threads_started":1}
Aug 10 01:47:22 dist-test-slave-kvfs krb5kdc[21019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM
I20260810 01:47:22.654444 22038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:22.644775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60362 (local address 127.9.77.190:46517)
0810 01:47:22.644961 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:22.644971 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:22.645112 (+ 141us) server_negotiation.cc:407] Connection header received
0810 01:47:22.645307 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:22.645310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:22.645361 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:22.645468 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:22.646129 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:22.646692 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:22.647567 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:22.647807 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:22.647926 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:22.650666 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:22.650669 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:22.650671 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:22.650698 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:22.652442 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:22.652930 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:22.652935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:22.652937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:22.653037 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:22.653364 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:22.653368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:22.653370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:22.653609 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:22.653781 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:22.654073 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 01:47:22.654228 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 01:47:22.657682 21094 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 01:47:22.658334 21094 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:47:22.664346 21534 tablet_service.cc:1467] Processing CreateTablet for tablet 9e24aa2845084ee1b1ce62b3bb8cc1b0 (DEFAULT_TABLE table=test-table [id=cbf38339384c40e6b244ef1bfbdb5f89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:47:22.664346 21313 tablet_service.cc:1467] Processing CreateTablet for tablet 9e24aa2845084ee1b1ce62b3bb8cc1b0 (DEFAULT_TABLE table=test-table [id=cbf38339384c40e6b244ef1bfbdb5f89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:47:22.664503 21751 tablet_service.cc:1467] Processing CreateTablet for tablet 9e24aa2845084ee1b1ce62b3bb8cc1b0 (DEFAULT_TABLE table=test-table [id=cbf38339384c40e6b244ef1bfbdb5f89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:47:22.664772 21534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e24aa2845084ee1b1ce62b3bb8cc1b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:22.664772 21313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e24aa2845084ee1b1ce62b3bb8cc1b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:22.664849 21751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e24aa2845084ee1b1ce62b3bb8cc1b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:22.668522 22046 tablet_bootstrap.cc:492] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P b89e2268ec7242c992c04fc4f68afdeb: Bootstrap starting.
I20260810 01:47:22.668771 22049 tablet_bootstrap.cc:492] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P a505fe46057f4a7da2c7ce013cb68638: Bootstrap starting.
I20260810 01:47:22.668859 22051 tablet_bootstrap.cc:492] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2: Bootstrap starting.
I20260810 01:47:22.669987 22051 tablet_bootstrap.cc:654] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:22.670336 22049 tablet_bootstrap.cc:654] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P a505fe46057f4a7da2c7ce013cb68638: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:22.670620 22046 tablet_bootstrap.cc:654] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P b89e2268ec7242c992c04fc4f68afdeb: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:22.671918 22049 tablet_bootstrap.cc:492] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P a505fe46057f4a7da2c7ce013cb68638: No bootstrap required, opened a new log
I20260810 01:47:22.672070 22049 ts_tablet_manager.cc:1397] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P a505fe46057f4a7da2c7ce013cb68638: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260810 01:47:22.672587 22049 raft_consensus.cc:348] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P a505fe46057f4a7da2c7ce013cb68638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.672729 22049 raft_consensus.cc:374] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P a505fe46057f4a7da2c7ce013cb68638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:22.672771 22049 raft_consensus.cc:729] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P a505fe46057f4a7da2c7ce013cb68638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505fe46057f4a7da2c7ce013cb68638, State: Initialized, Role: FOLLOWER
I20260810 01:47:22.672947 22049 consensus_queue.cc:260] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P a505fe46057f4a7da2c7ce013cb68638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.673274 22049 ts_tablet_manager.cc:1428] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P a505fe46057f4a7da2c7ce013cb68638: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 01:47:22.674575 22046 tablet_bootstrap.cc:492] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P b89e2268ec7242c992c04fc4f68afdeb: No bootstrap required, opened a new log
I20260810 01:47:22.674739 22046 ts_tablet_manager.cc:1397] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P b89e2268ec7242c992c04fc4f68afdeb: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260810 01:47:22.675405 22046 raft_consensus.cc:348] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P b89e2268ec7242c992c04fc4f68afdeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.675549 22046 raft_consensus.cc:374] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P b89e2268ec7242c992c04fc4f68afdeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:22.675587 22046 raft_consensus.cc:729] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P b89e2268ec7242c992c04fc4f68afdeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89e2268ec7242c992c04fc4f68afdeb, State: Initialized, Role: FOLLOWER
I20260810 01:47:22.675752 22046 consensus_queue.cc:260] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P b89e2268ec7242c992c04fc4f68afdeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.676616 22046 ts_tablet_manager.cc:1428] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P b89e2268ec7242c992c04fc4f68afdeb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 01:47:22.680291 22051 tablet_bootstrap.cc:492] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2: No bootstrap required, opened a new log
I20260810 01:47:22.680445 22051 ts_tablet_manager.cc:1397] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260810 01:47:22.680966 22051 raft_consensus.cc:348] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.681092 22051 raft_consensus.cc:374] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:22.681157 22051 raft_consensus.cc:729] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86a959ed6f54447bf5c79b8dc35f9f2, State: Initialized, Role: FOLLOWER
I20260810 01:47:22.681344 22051 consensus_queue.cc:260] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.681695 22051 ts_tablet_manager.cc:1428] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 01:47:22.761088 22011 raft_consensus.cc:482] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:47:22.761310 22011 raft_consensus.cc:504] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.761821 22011 leader_election.cc:288] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a505fe46057f4a7da2c7ce013cb68638 (127.9.77.129:41641), b89e2268ec7242c992c04fc4f68afdeb (127.9.77.130:44931)
I20260810 01:47:22.762637 21564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e24aa2845084ee1b1ce62b3bb8cc1b0" candidate_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89e2268ec7242c992c04fc4f68afdeb" is_pre_election: true
I20260810 01:47:22.762676 21342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e24aa2845084ee1b1ce62b3bb8cc1b0" candidate_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505fe46057f4a7da2c7ce013cb68638" is_pre_election: true
I20260810 01:47:22.762878 21342 raft_consensus.cc:2454] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P a505fe46057f4a7da2c7ce013cb68638 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86a959ed6f54447bf5c79b8dc35f9f2 in term 0.
I20260810 01:47:22.762893 21564 raft_consensus.cc:2454] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P b89e2268ec7242c992c04fc4f68afdeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86a959ed6f54447bf5c79b8dc35f9f2 in term 0.
I20260810 01:47:22.763355 21679 leader_election.cc:302] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a505fe46057f4a7da2c7ce013cb68638, c86a959ed6f54447bf5c79b8dc35f9f2; no voters:
I20260810 01:47:22.763635 22011 raft_consensus.cc:2781] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:47:22.763710 22011 raft_consensus.cc:482] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:47:22.763762 22011 raft_consensus.cc:3037] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:22.765182 22011 raft_consensus.cc:504] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.765599 22011 leader_election.cc:288] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2 [CANDIDATE]: Term 1 election: Requested vote from peers a505fe46057f4a7da2c7ce013cb68638 (127.9.77.129:41641), b89e2268ec7242c992c04fc4f68afdeb (127.9.77.130:44931)
I20260810 01:47:22.766132 21342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e24aa2845084ee1b1ce62b3bb8cc1b0" candidate_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505fe46057f4a7da2c7ce013cb68638"
I20260810 01:47:22.766206 21564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e24aa2845084ee1b1ce62b3bb8cc1b0" candidate_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89e2268ec7242c992c04fc4f68afdeb"
I20260810 01:47:22.766322 21342 raft_consensus.cc:3037] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P a505fe46057f4a7da2c7ce013cb68638 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:22.766418 21564 raft_consensus.cc:3037] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P b89e2268ec7242c992c04fc4f68afdeb [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:22.767992 21342 raft_consensus.cc:2454] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P a505fe46057f4a7da2c7ce013cb68638 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86a959ed6f54447bf5c79b8dc35f9f2 in term 1.
I20260810 01:47:22.768386 21564 raft_consensus.cc:2454] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P b89e2268ec7242c992c04fc4f68afdeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86a959ed6f54447bf5c79b8dc35f9f2 in term 1.
I20260810 01:47:22.768481 21679 leader_election.cc:302] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a505fe46057f4a7da2c7ce013cb68638, c86a959ed6f54447bf5c79b8dc35f9f2; no voters:
I20260810 01:47:22.768738 22011 raft_consensus.cc:2781] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:47:22.768857 22011 raft_consensus.cc:686] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 1 LEADER]: Becoming Leader. State: Replica: c86a959ed6f54447bf5c79b8dc35f9f2, State: Running, Role: LEADER
I20260810 01:47:22.769066 22011 consensus_queue.cc:237] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.771224 21094 catalog_manager.cc:5495] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86a959ed6f54447bf5c79b8dc35f9f2 (127.9.77.131). New cstate: current_term: 1 leader_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } health_report { overall_health: HEALTHY } } }
I20260810 01:47:22.801971 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:22.798309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60706 (local address 127.9.77.131:35369)
0810 01:47:22.798478 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:22.798484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:22.798502 (+ 18us) server_negotiation.cc:407] Connection header received
0810 01:47:22.798608 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:22.798616 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:22.798671 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:22.798767 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:47:22.799069 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:22.799641 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:22.800764 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:22.801018 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:22.801139 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:22.801446 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:47:22.801543 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:47:22.801628 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:22.801723 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 01:47:22.801782 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 01:47:22.804539 21564 raft_consensus.cc:1261] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P b89e2268ec7242c992c04fc4f68afdeb [term 1 FOLLOWER]: Refusing update from remote peer c86a959ed6f54447bf5c79b8dc35f9f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:47:22.804596 21342 raft_consensus.cc:1261] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P a505fe46057f4a7da2c7ce013cb68638 [term 1 FOLLOWER]: Refusing update from remote peer c86a959ed6f54447bf5c79b8dc35f9f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:47:22.805104 21922 consensus_queue.cc:1035] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:47:22.805315 22011 consensus_queue.cc:1035] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:47:22.813164 21092 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:47:22.813338 21092 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:47:22.815697 21092 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fab89cac2fe444da5e6f375fef7e249: Sending DeleteTablet for 3 replicas of tablet 9e24aa2845084ee1b1ce62b3bb8cc1b0
I20260810 01:47:22.816391 21751 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e24aa2845084ee1b1ce62b3bb8cc1b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:47:22 UTC) from {username='oryx', principal='oryx/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:60672
I20260810 01:47:22.816478 21313 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e24aa2845084ee1b1ce62b3bb8cc1b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:47:22 UTC) from {username='oryx', principal='oryx/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:50576
I20260810 01:47:22.816629 22022 tablet_replica.cc:330] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2: stopping tablet replica
I20260810 01:47:22.816685 22027 tablet_replica.cc:330] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P a505fe46057f4a7da2c7ce013cb68638: stopping tablet replica
I20260810 01:47:22.816798 22027 raft_consensus.cc:2229] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P a505fe46057f4a7da2c7ce013cb68638 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:47:22.816780 22022 raft_consensus.cc:2229] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 1 LEADER]: Raft consensus shutting down.
I20260810 01:47:22.816957 22027 raft_consensus.cc:2258] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P a505fe46057f4a7da2c7ce013cb68638 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:47:22.817013 22022 raft_consensus.cc:2258] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:47:22.817129 21092 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 01:47:22.817395 21534 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e24aa2845084ee1b1ce62b3bb8cc1b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:47:22 UTC) from {username='oryx', principal='oryx/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:60000
W20260810 01:47:22.817655 21092 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:47:22.817966 22027 ts_tablet_manager.cc:1905] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P a505fe46057f4a7da2c7ce013cb68638: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:47:22.818477 22028 tablet_replica.cc:330] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P b89e2268ec7242c992c04fc4f68afdeb: stopping tablet replica
I20260810 01:47:22.818614 22028 raft_consensus.cc:2229] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P b89e2268ec7242c992c04fc4f68afdeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:47:22.818746 22028 raft_consensus.cc:2258] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P b89e2268ec7242c992c04fc4f68afdeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:47:22.819543 22028 ts_tablet_manager.cc:1905] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P b89e2268ec7242c992c04fc4f68afdeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:47:22.820577 22022 ts_tablet_manager.cc:1905] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:47:22.821092 22027 ts_tablet_manager.cc:1918] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P a505fe46057f4a7da2c7ce013cb68638: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:47:22.821179 22027 log.cc:1196] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P a505fe46057f4a7da2c7ce013cb68638: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-0/wal/wals/9e24aa2845084ee1b1ce62b3bb8cc1b0
I20260810 01:47:22.821485 22027 ts_tablet_manager.cc:1939] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P a505fe46057f4a7da2c7ce013cb68638: Deleting consensus metadata
I20260810 01:47:22.822321 21073 catalog_manager.cc:4841] TS a505fe46057f4a7da2c7ce013cb68638 (127.9.77.129:41641): tablet 9e24aa2845084ee1b1ce62b3bb8cc1b0 (table test-table [id=cbf38339384c40e6b244ef1bfbdb5f89]) successfully deleted
I20260810 01:47:22.822532 22028 ts_tablet_manager.cc:1918] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P b89e2268ec7242c992c04fc4f68afdeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:47:22.822613 22028 log.cc:1196] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P b89e2268ec7242c992c04fc4f68afdeb: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-1/wal/wals/9e24aa2845084ee1b1ce62b3bb8cc1b0
I20260810 01:47:22.822914 22028 ts_tablet_manager.cc:1939] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P b89e2268ec7242c992c04fc4f68afdeb: Deleting consensus metadata
I20260810 01:47:22.823246 21534 tablet_service.cc:1467] Processing CreateTablet for tablet 2f7d090e93e949e5912aa4c63d08d52a (DEFAULT_TABLE table=test-table [id=534e10a00b8b4386ad7d478fb4624a08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:47:22.823441 21751 tablet_service.cc:1467] Processing CreateTablet for tablet 2f7d090e93e949e5912aa4c63d08d52a (DEFAULT_TABLE table=test-table [id=534e10a00b8b4386ad7d478fb4624a08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:47:22.823526 21534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f7d090e93e949e5912aa4c63d08d52a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:22.823611 21073 catalog_manager.cc:4841] TS b89e2268ec7242c992c04fc4f68afdeb (127.9.77.130:44931): tablet 9e24aa2845084ee1b1ce62b3bb8cc1b0 (table test-table [id=cbf38339384c40e6b244ef1bfbdb5f89]) successfully deleted
I20260810 01:47:22.823654 22022 ts_tablet_manager.cc:1918] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:47:22.823722 21751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f7d090e93e949e5912aa4c63d08d52a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:22.823721 22022 log.cc:1196] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-2/wal/wals/9e24aa2845084ee1b1ce62b3bb8cc1b0
I20260810 01:47:22.824707 22022 ts_tablet_manager.cc:1939] T 9e24aa2845084ee1b1ce62b3bb8cc1b0 P c86a959ed6f54447bf5c79b8dc35f9f2: Deleting consensus metadata
I20260810 01:47:22.825544 21075 catalog_manager.cc:4841] TS c86a959ed6f54447bf5c79b8dc35f9f2 (127.9.77.131:35369): tablet 9e24aa2845084ee1b1ce62b3bb8cc1b0 (table test-table [id=cbf38339384c40e6b244ef1bfbdb5f89]) successfully deleted
I20260810 01:47:22.825978 22046 tablet_bootstrap.cc:492] T 2f7d090e93e949e5912aa4c63d08d52a P b89e2268ec7242c992c04fc4f68afdeb: Bootstrap starting.
I20260810 01:47:22.826045 22051 tablet_bootstrap.cc:492] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2: Bootstrap starting.
I20260810 01:47:22.826622 21313 tablet_service.cc:1467] Processing CreateTablet for tablet 2f7d090e93e949e5912aa4c63d08d52a (DEFAULT_TABLE table=test-table [id=534e10a00b8b4386ad7d478fb4624a08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:47:22.826906 21313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f7d090e93e949e5912aa4c63d08d52a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:22.826927 22051 tablet_bootstrap.cc:654] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:22.827080 22046 tablet_bootstrap.cc:654] T 2f7d090e93e949e5912aa4c63d08d52a P b89e2268ec7242c992c04fc4f68afdeb: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:22.828565 22046 tablet_bootstrap.cc:492] T 2f7d090e93e949e5912aa4c63d08d52a P b89e2268ec7242c992c04fc4f68afdeb: No bootstrap required, opened a new log
I20260810 01:47:22.828639 22046 ts_tablet_manager.cc:1397] T 2f7d090e93e949e5912aa4c63d08d52a P b89e2268ec7242c992c04fc4f68afdeb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 01:47:22.828814 22051 tablet_bootstrap.cc:492] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2: No bootstrap required, opened a new log
I20260810 01:47:22.828892 22051 ts_tablet_manager.cc:1397] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 01:47:22.829031 22046 raft_consensus.cc:348] T 2f7d090e93e949e5912aa4c63d08d52a P b89e2268ec7242c992c04fc4f68afdeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.829125 22046 raft_consensus.cc:374] T 2f7d090e93e949e5912aa4c63d08d52a P b89e2268ec7242c992c04fc4f68afdeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:22.829146 22046 raft_consensus.cc:729] T 2f7d090e93e949e5912aa4c63d08d52a P b89e2268ec7242c992c04fc4f68afdeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89e2268ec7242c992c04fc4f68afdeb, State: Initialized, Role: FOLLOWER
I20260810 01:47:22.829313 22051 raft_consensus.cc:348] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.829327 22046 consensus_queue.cc:260] T 2f7d090e93e949e5912aa4c63d08d52a P b89e2268ec7242c992c04fc4f68afdeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.829447 22051 raft_consensus.cc:374] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:22.829484 22051 raft_consensus.cc:729] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86a959ed6f54447bf5c79b8dc35f9f2, State: Initialized, Role: FOLLOWER
I20260810 01:47:22.829641 22049 tablet_bootstrap.cc:492] T 2f7d090e93e949e5912aa4c63d08d52a P a505fe46057f4a7da2c7ce013cb68638: Bootstrap starting.
I20260810 01:47:22.829665 22046 ts_tablet_manager.cc:1428] T 2f7d090e93e949e5912aa4c63d08d52a P b89e2268ec7242c992c04fc4f68afdeb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 01:47:22.829615 22051 consensus_queue.cc:260] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.829901 22051 ts_tablet_manager.cc:1428] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 01:47:22.830714 22049 tablet_bootstrap.cc:654] T 2f7d090e93e949e5912aa4c63d08d52a P a505fe46057f4a7da2c7ce013cb68638: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:22.832126 22049 tablet_bootstrap.cc:492] T 2f7d090e93e949e5912aa4c63d08d52a P a505fe46057f4a7da2c7ce013cb68638: No bootstrap required, opened a new log
I20260810 01:47:22.832254 22049 ts_tablet_manager.cc:1397] T 2f7d090e93e949e5912aa4c63d08d52a P a505fe46057f4a7da2c7ce013cb68638: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 01:47:22.832792 22049 raft_consensus.cc:348] T 2f7d090e93e949e5912aa4c63d08d52a P a505fe46057f4a7da2c7ce013cb68638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.832928 22049 raft_consensus.cc:374] T 2f7d090e93e949e5912aa4c63d08d52a P a505fe46057f4a7da2c7ce013cb68638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:22.832981 22049 raft_consensus.cc:729] T 2f7d090e93e949e5912aa4c63d08d52a P a505fe46057f4a7da2c7ce013cb68638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505fe46057f4a7da2c7ce013cb68638, State: Initialized, Role: FOLLOWER
I20260810 01:47:22.833179 22049 consensus_queue.cc:260] T 2f7d090e93e949e5912aa4c63d08d52a P a505fe46057f4a7da2c7ce013cb68638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.833492 22049 ts_tablet_manager.cc:1428] T 2f7d090e93e949e5912aa4c63d08d52a P a505fe46057f4a7da2c7ce013cb68638: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 01:47:22.945990 21929 raft_consensus.cc:482] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:47:22.946237 21929 raft_consensus.cc:504] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.946800 21929 leader_election.cc:288] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a505fe46057f4a7da2c7ce013cb68638 (127.9.77.129:41641), b89e2268ec7242c992c04fc4f68afdeb (127.9.77.130:44931)
I20260810 01:47:22.947535 21342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7d090e93e949e5912aa4c63d08d52a" candidate_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505fe46057f4a7da2c7ce013cb68638" is_pre_election: true
I20260810 01:47:22.947582 21564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7d090e93e949e5912aa4c63d08d52a" candidate_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89e2268ec7242c992c04fc4f68afdeb" is_pre_election: true
I20260810 01:47:22.947758 21342 raft_consensus.cc:2454] T 2f7d090e93e949e5912aa4c63d08d52a P a505fe46057f4a7da2c7ce013cb68638 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86a959ed6f54447bf5c79b8dc35f9f2 in term 0.
I20260810 01:47:22.947785 21564 raft_consensus.cc:2454] T 2f7d090e93e949e5912aa4c63d08d52a P b89e2268ec7242c992c04fc4f68afdeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86a959ed6f54447bf5c79b8dc35f9f2 in term 0.
I20260810 01:47:22.948280 21679 leader_election.cc:302] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b89e2268ec7242c992c04fc4f68afdeb, c86a959ed6f54447bf5c79b8dc35f9f2; no voters:
I20260810 01:47:22.948525 21929 raft_consensus.cc:2781] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:47:22.948608 21929 raft_consensus.cc:482] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:47:22.948662 21929 raft_consensus.cc:3037] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:22.950188 21929 raft_consensus.cc:504] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.950582 21929 leader_election.cc:288] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2 [CANDIDATE]: Term 1 election: Requested vote from peers a505fe46057f4a7da2c7ce013cb68638 (127.9.77.129:41641), b89e2268ec7242c992c04fc4f68afdeb (127.9.77.130:44931)
I20260810 01:47:22.951058 21342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7d090e93e949e5912aa4c63d08d52a" candidate_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a505fe46057f4a7da2c7ce013cb68638"
I20260810 01:47:22.951133 21564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7d090e93e949e5912aa4c63d08d52a" candidate_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89e2268ec7242c992c04fc4f68afdeb"
I20260810 01:47:22.951206 21342 raft_consensus.cc:3037] T 2f7d090e93e949e5912aa4c63d08d52a P a505fe46057f4a7da2c7ce013cb68638 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:22.951325 21564 raft_consensus.cc:3037] T 2f7d090e93e949e5912aa4c63d08d52a P b89e2268ec7242c992c04fc4f68afdeb [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:22.952479 21564 raft_consensus.cc:2454] T 2f7d090e93e949e5912aa4c63d08d52a P b89e2268ec7242c992c04fc4f68afdeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86a959ed6f54447bf5c79b8dc35f9f2 in term 1.
I20260810 01:47:22.952499 21342 raft_consensus.cc:2454] T 2f7d090e93e949e5912aa4c63d08d52a P a505fe46057f4a7da2c7ce013cb68638 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86a959ed6f54447bf5c79b8dc35f9f2 in term 1.
I20260810 01:47:22.952812 21679 leader_election.cc:302] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b89e2268ec7242c992c04fc4f68afdeb, c86a959ed6f54447bf5c79b8dc35f9f2; no voters:
I20260810 01:47:22.953042 21929 raft_consensus.cc:2781] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:47:22.953159 21929 raft_consensus.cc:686] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2 [term 1 LEADER]: Becoming Leader. State: Replica: c86a959ed6f54447bf5c79b8dc35f9f2, State: Running, Role: LEADER
I20260810 01:47:22.953331 21929 consensus_queue.cc:237] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } }
I20260810 01:47:22.955469 21094 catalog_manager.cc:5495] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86a959ed6f54447bf5c79b8dc35f9f2 (127.9.77.131). New cstate: current_term: 1 leader_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86a959ed6f54447bf5c79b8dc35f9f2" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35369 } health_report { overall_health: HEALTHY } } }
I20260810 01:47:23.002976 21564 raft_consensus.cc:1261] T 2f7d090e93e949e5912aa4c63d08d52a P b89e2268ec7242c992c04fc4f68afdeb [term 1 FOLLOWER]: Refusing update from remote peer c86a959ed6f54447bf5c79b8dc35f9f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:47:23.003090 21342 raft_consensus.cc:1261] T 2f7d090e93e949e5912aa4c63d08d52a P a505fe46057f4a7da2c7ce013cb68638 [term 1 FOLLOWER]: Refusing update from remote peer c86a959ed6f54447bf5c79b8dc35f9f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:47:23.003504 21929 consensus_queue.cc:1035] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89e2268ec7242c992c04fc4f68afdeb" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:47:23.003690 21922 consensus_queue.cc:1035] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a505fe46057f4a7da2c7ce013cb68638" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 41641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:47:23.038673 21094 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:47:23.038901 21094 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:47:23.041350 21094 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fab89cac2fe444da5e6f375fef7e249: Sending DeleteTablet for 3 replicas of tablet 2f7d090e93e949e5912aa4c63d08d52a
I20260810 01:47:23.042017 21751 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f7d090e93e949e5912aa4c63d08d52a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:47:23 UTC) from {username='oryx', principal='oryx/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:60672
I20260810 01:47:23.042256 21534 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f7d090e93e949e5912aa4c63d08d52a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:47:23 UTC) from {username='oryx', principal='oryx/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:60000
I20260810 01:47:23.042404 21313 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f7d090e93e949e5912aa4c63d08d52a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:47:23 UTC) from {username='oryx', principal='oryx/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:50576
I20260810 01:47:23.042510 22022 tablet_replica.cc:330] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2: stopping tablet replica
I20260810 01:47:23.042659 22022 raft_consensus.cc:2229] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2 [term 1 LEADER]: Raft consensus shutting down.
I20260810 01:47:23.042878 22022 raft_consensus.cc:2258] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:47:23.043110 22028 tablet_replica.cc:330] T 2f7d090e93e949e5912aa4c63d08d52a P b89e2268ec7242c992c04fc4f68afdeb: stopping tablet replica
I20260810 01:47:23.043257 22028 raft_consensus.cc:2229] T 2f7d090e93e949e5912aa4c63d08d52a P b89e2268ec7242c992c04fc4f68afdeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:47:23.043417 22028 raft_consensus.cc:2258] T 2f7d090e93e949e5912aa4c63d08d52a P b89e2268ec7242c992c04fc4f68afdeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:47:23.043751 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 21209
I20260810 01:47:23.043848 22027 tablet_replica.cc:330] T 2f7d090e93e949e5912aa4c63d08d52a P a505fe46057f4a7da2c7ce013cb68638: stopping tablet replica
I20260810 01:47:23.044004 22027 raft_consensus.cc:2229] T 2f7d090e93e949e5912aa4c63d08d52a P a505fe46057f4a7da2c7ce013cb68638 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:47:23.044155 22027 raft_consensus.cc:2258] T 2f7d090e93e949e5912aa4c63d08d52a P a505fe46057f4a7da2c7ce013cb68638 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:47:23.048007 22028 ts_tablet_manager.cc:1905] T 2f7d090e93e949e5912aa4c63d08d52a P b89e2268ec7242c992c04fc4f68afdeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:47:23.050747 22028 ts_tablet_manager.cc:1918] T 2f7d090e93e949e5912aa4c63d08d52a P b89e2268ec7242c992c04fc4f68afdeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 01:47:23.050832 22028 log.cc:1196] T 2f7d090e93e949e5912aa4c63d08d52a P b89e2268ec7242c992c04fc4f68afdeb: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-1/wal/wals/2f7d090e93e949e5912aa4c63d08d52a
I20260810 01:47:23.051126 22028 ts_tablet_manager.cc:1939] T 2f7d090e93e949e5912aa4c63d08d52a P b89e2268ec7242c992c04fc4f68afdeb: Deleting consensus metadata
I20260810 01:47:23.054683 22022 ts_tablet_manager.cc:1905] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:47:23.055037 21073 catalog_manager.cc:4841] TS b89e2268ec7242c992c04fc4f68afdeb (127.9.77.130:44931): tablet 2f7d090e93e949e5912aa4c63d08d52a (table test-table [id=534e10a00b8b4386ad7d478fb4624a08]) successfully deleted
I20260810 01:47:23.057888 22022 ts_tablet_manager.cc:1918] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 01:47:23.058010 22022 log.cc:1196] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786326392877349-9526-0/minicluster-data/ts-2/wal/wals/2f7d090e93e949e5912aa4c63d08d52a
I20260810 01:47:23.058354 22022 ts_tablet_manager.cc:1939] T 2f7d090e93e949e5912aa4c63d08d52a P c86a959ed6f54447bf5c79b8dc35f9f2: Deleting consensus metadata
I20260810 01:47:23.059257 21075 catalog_manager.cc:4841] TS c86a959ed6f54447bf5c79b8dc35f9f2 (127.9.77.131:35369): tablet 2f7d090e93e949e5912aa4c63d08d52a (table test-table [id=534e10a00b8b4386ad7d478fb4624a08]) successfully deleted
W20260810 01:47:23.063758 21073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.77.129:41641 (error 108)
I20260810 01:47:23.064100 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 21433
I20260810 01:47:23.069748 22074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:23.067984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:41641 (local address 127.0.0.1:50584)
0810 01:47:23.069524 (+ 1540us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:23.069624 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.129:41641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1418,"thread_start_us":1012,"threads_started":1}
W20260810 01:47:23.069921 21073 catalog_manager.cc:4584] TS a505fe46057f4a7da2c7ce013cb68638 (127.9.77.129:41641): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f7d090e93e949e5912aa4c63d08d52a: Network error: Client connection negotiation failed: client connection to 127.9.77.129:41641: connect: Connection refused (error 111)
I20260810 01:47:23.082468 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 21653
I20260810 01:47:23.099251 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 21044
2026-08-10T01:47:23Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (4167 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:47:23 dist-test-slave-kvfs krb5kdc[22081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:47:23 dist-test-slave-kvfs krb5kdc[22081](info): set up 2 sockets
Aug 10 01:47:23 dist-test-slave-kvfs krb5kdc[22081](info): commencing operation
krb5kdc: starting...
W20260810 01:47:24.417600 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.280s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:47:24 dist-test-slave-kvfs krb5kdc[22081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:47:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:47:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:47:24.568727 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:45101
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:34343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:45101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:24.705197 22103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:24.705634 22103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:24.705717 22103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:24.709959 22103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:47:24.710069 22103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:24.710106 22103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:24.710145 22103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:47:24.710204 22103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:47:24.715983 22103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:34343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:45101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:45101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:24.717531 22103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:24.718914 22103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:24.729525 22115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:24.729481 22117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:24.729471 22114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:24.729983 22103 server_base.cc:1034] running on GCE node
I20260810 01:47:24.730723 22103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:24.732168 22103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:24.733364 22103 hybrid_clock.cc:648] HybridClock initialized: now 1786326444733351 us; error 38 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T01:47:24Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1636 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:47:24 dist-test-slave-kvfs krb5kdc[22123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:47:24 dist-test-slave-kvfs krb5kdc[22123](info): set up 2 sockets
Aug 10 01:47:24 dist-test-slave-kvfs krb5kdc[22123](info): commencing operation
krb5kdc: starting...
W20260810 01:47:26.111162 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.337s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:47:26 dist-test-slave-kvfs krb5kdc[22123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:47:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:47:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:47:26.271903 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:45897
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:45897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:26.412626 22142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:26.412995 22142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:26.413040 22142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:26.417217 22142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:47:26.417289 22142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:26.417310 22142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:26.417328 22142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:47:26.417343 22142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:47:26.423275 22142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:45897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:45897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:26.424752 22142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:26.426085 22142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:26.436368 22149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:26.436390 22153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:26.436474 22151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:26.436666 22142 server_base.cc:1034] running on GCE node
I20260810 01:47:26.437512 22142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:26.438937 22142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:26.440117 22142 hybrid_clock.cc:648] HybridClock initialized: now 1786326446440085 us; error 48 us; skew 500 ppm
Aug 10 01:47:26 dist-test-slave-kvfs krb5kdc[22123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326446, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:26.443871 22142 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:47:26.445101 22142 webserver.cc:460] Webserver started at http://127.9.77.190:41135/ using document root <none> and password file <none>
I20260810 01:47:26.445847 22142 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:26.445920 22142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:26.446210 22142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:26.448901 22142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "0574e6491e784c879226221e57c30c0c"
format_stamp: "Formatted at 2026-08-10 01:47:26 on dist-test-slave-kvfs"
server_key: "32d23e93fa0f2404fddf78eec3e9df1c"
server_key_iv: "6c289cdb50705b31970314e86906a8f1"
server_key_version: "encryptionkey@0"
I20260810 01:47:26.449584 22142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "0574e6491e784c879226221e57c30c0c"
format_stamp: "Formatted at 2026-08-10 01:47:26 on dist-test-slave-kvfs"
server_key: "32d23e93fa0f2404fddf78eec3e9df1c"
server_key_iv: "6c289cdb50705b31970314e86906a8f1"
server_key_version: "encryptionkey@0"
I20260810 01:47:26.454250 22142 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 01:47:26.457239 22164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:26.458827 22142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:47:26.459044 22142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "0574e6491e784c879226221e57c30c0c"
format_stamp: "Formatted at 2026-08-10 01:47:26 on dist-test-slave-kvfs"
server_key: "32d23e93fa0f2404fddf78eec3e9df1c"
server_key_iv: "6c289cdb50705b31970314e86906a8f1"
server_key_version: "encryptionkey@0"
I20260810 01:47:26.459187 22142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:26.482717 22142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:26.487172 22142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:26.487479 22142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:26.496681 22142 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:45897
I20260810 01:47:26.496709 22245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:45897 every 8 connection(s)
I20260810 01:47:26.498015 22142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:47:26.501735 22247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:26.502709 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 22142
I20260810 01:47:26.502885 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:47:26.503309 9526 external_mini_cluster.cc:1428] Setting key 18f814b9d0250e2ed7f552c4e9c3f536
I20260810 01:47:26.509845 22247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c: Bootstrap starting.
I20260810 01:47:26.512952 22247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c: Neither blocks nor log segments found. Creating new log.
Aug 10 01:47:26 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:47:26.514127 22247 log.cc:824] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:26.517159 22247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c: No bootstrap required, opened a new log
I20260810 01:47:26.520994 22252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:26.505319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33964 (local address 127.9.77.190:45897)
0810 01:47:26.505935 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:26.505953 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:26.506000 (+ 47us) server_negotiation.cc:407] Connection header received
0810 01:47:26.506782 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:26.506804 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:26.507218 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:26.507638 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:26.508569 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:26.509657 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:26.510744 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:26.511285 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:26.511476 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:26.514801 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:26.514806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:26.514822 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:26.514851 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:26.517137 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:26.517693 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:26.517699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:26.517706 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:26.517866 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:26.518159 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:26.518163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:26.518165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:26.518768 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:26.519024 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:26.519231 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 01:47:26.519621 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":189,"threads_started":1}
I20260810 01:47:26.522670 22247 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0574e6491e784c879226221e57c30c0c" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45897 } }
I20260810 01:47:26.522945 22247 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:26.523011 22247 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0574e6491e784c879226221e57c30c0c, State: Initialized, Role: FOLLOWER
I20260810 01:47:26.523622 22247 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0574e6491e784c879226221e57c30c0c" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45897 } }
I20260810 01:47:26.523836 22247 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:47:26.523950 22247 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:47:26.524111 22247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:26.525279 22247 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0574e6491e784c879226221e57c30c0c" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45897 } }
I20260810 01:47:26.525718 22247 leader_election.cc:302] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0574e6491e784c879226221e57c30c0c; no voters:
I20260810 01:47:26.526113 22247 leader_election.cc:288] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:47:26.526262 22254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:47:26.526530 22254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [term 1 LEADER]: Becoming Leader. State: Replica: 0574e6491e784c879226221e57c30c0c, State: Running, Role: LEADER
I20260810 01:47:26.526937 22254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0574e6491e784c879226221e57c30c0c" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45897 } }
I20260810 01:47:26.527614 22247 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:47:26.529081 22258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0574e6491e784c879226221e57c30c0c. Latest consensus state: current_term: 1 leader_uuid: "0574e6491e784c879226221e57c30c0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0574e6491e784c879226221e57c30c0c" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45897 } } }
I20260810 01:47:26.529064 22256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0574e6491e784c879226221e57c30c0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0574e6491e784c879226221e57c30c0c" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45897 } } }
I20260810 01:47:26.529229 22258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:26.529810 22271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:47:26.529237 22256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:26.533411 22271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:47:26.541153 22271 catalog_manager.cc:1347] Generated new cluster ID: d3c75ad327914bc38ac97da24f5f6dd7
I20260810 01:47:26.541262 22271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:47:26.557565 22271 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:47:26.558862 22271 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:47:26.570798 22271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c: Generated new TSK 0
I20260810 01:47:26.572160 22271 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.77.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
I20260810 01:47:26.620709 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:0
--local_ip_for_outbound_sockets=127.9.77.129
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:45897
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:26.768321 22292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:26.768679 22292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:26.768728 22292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:26.773133 22292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:26.773206 22292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:26.773300 22292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:47:26.779237 22292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:45897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:26.780767 22292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:26.782131 22292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:26.794525 22302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:26.794523 22298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:26.794544 22299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:26.794937 22292 server_base.cc:1034] running on GCE node
I20260810 01:47:26.795457 22292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:26.796409 22292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:26.797631 22292 hybrid_clock.cc:648] HybridClock initialized: now 1786326446797601 us; error 47 us; skew 500 ppm
Aug 10 01:47:26 dist-test-slave-kvfs krb5kdc[22123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326446, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:26.801313 22292 init.cc:377] Logged in from keytab as kudu/127.9.77.129@KRBTEST.COM (short username kudu)
I20260810 01:47:26.802542 22292 webserver.cc:460] Webserver started at http://127.9.77.129:44029/ using document root <none> and password file <none>
I20260810 01:47:26.803328 22292 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:26.803429 22292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:26.803720 22292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:26.806504 22292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/data/instance:
uuid: "1edc4ceb962f467dbbe10a604a8eb62e"
format_stamp: "Formatted at 2026-08-10 01:47:26 on dist-test-slave-kvfs"
server_key: "dee5d24c0862dfb257abdf007e2499aa"
server_key_iv: "fd94970d1938a5e3f0f9f8acc8033456"
server_key_version: "encryptionkey@0"
I20260810 01:47:26.807236 22292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance:
uuid: "1edc4ceb962f467dbbe10a604a8eb62e"
format_stamp: "Formatted at 2026-08-10 01:47:26 on dist-test-slave-kvfs"
server_key: "dee5d24c0862dfb257abdf007e2499aa"
server_key_iv: "fd94970d1938a5e3f0f9f8acc8033456"
server_key_version: "encryptionkey@0"
I20260810 01:47:26.812230 22292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260810 01:47:26.815402 22314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:26.817065 22292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:47:26.817234 22292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "1edc4ceb962f467dbbe10a604a8eb62e"
format_stamp: "Formatted at 2026-08-10 01:47:26 on dist-test-slave-kvfs"
server_key: "dee5d24c0862dfb257abdf007e2499aa"
server_key_iv: "fd94970d1938a5e3f0f9f8acc8033456"
server_key_version: "encryptionkey@0"
I20260810 01:47:26.817387 22292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:26.834514 22292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:26.838575 22292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:26.838845 22292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:26.839638 22292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:26.840792 22292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:26.840869 22292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:26.840960 22292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:26.841005 22292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:26.851982 22292 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:37281
I20260810 01:47:26.852005 22518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:37281 every 8 connection(s)
I20260810 01:47:26.853190 22292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:47:26.862986 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 22292
I20260810 01:47:26.863157 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance
I20260810 01:47:26.863615 9526 external_mini_cluster.cc:1428] Setting key f4cff8662248f5987d81f52a540eb380
Aug 10 01:47:26 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326446, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:47:26.869653 22252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:26.855533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:46397 (local address 127.9.77.190:45897)
0810 01:47:26.855725 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:26.855730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:26.857104 (+ 1374us) server_negotiation.cc:407] Connection header received
0810 01:47:26.858234 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:26.858238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:26.858314 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:26.858433 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:26.859913 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:26.860612 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:26.861912 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:26.862185 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:26.862309 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:26.865548 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:26.865552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:26.865555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:26.865594 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:26.867196 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:26.867803 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:26.867807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:26.867810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:26.867908 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:26.868325 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:26.868329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:26.868331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:26.868554 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:26.868678 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:26.869296 (+ 618us) server_negotiation.cc:299] Negotiation successful
0810 01:47:26.869459 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 01:47:26.870759 22524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:26.856014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.129:46397)
0810 01:47:26.856908 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:26.856965 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:26.857974 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:26.858657 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:26.858665 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:26.859120 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:26.859730 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:26.859746 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:26.860781 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:26.860784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:26.861773 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:26.861783 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:26.862381 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:26.862385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:26.862610 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:26.863430 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:26.863453 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:26.865346 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:26.867345 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:26.867352 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:26.867372 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:26.867689 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:26.868013 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:26.868041 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:26.868046 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:26.868218 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:26.868688 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:26.868698 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:26.869018 (+ 320us) client_negotiation.cc:769] Sending connection context
0810 01:47:26.869236 (+ 218us) client_negotiation.cc:240] Negotiation successful
0810 01:47:26.869657 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":146,"threads_started":1}
I20260810 01:47:26.872502 22519 heartbeater.cc:347] Connected to a master server at 127.9.77.190:45897
I20260810 01:47:26.872852 22519 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:26.873678 22519 heartbeater.cc:511] Master 127.9.77.190:45897 requested a full tablet report, sending...
I20260810 01:47:26.875759 22189 ts_manager.cc:194] Registered new tserver with Master: 1edc4ceb962f467dbbe10a604a8eb62e (127.9.77.129:37281)
WARNING: no policy specified for kudu/127.9.77.130@KRBTEST.COM; defaulting to no policy
I20260810 01:47:26.877283 22189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.129@KRBTEST.COM'} at 127.9.77.129:46397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
I20260810 01:47:26.900527 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:0
--local_ip_for_outbound_sockets=127.9.77.130
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:45897
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:27.038240 22534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:27.038651 22534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:27.038733 22534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:27.043195 22534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:27.043315 22534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:27.043424 22534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:47:27.049540 22534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:45897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:27.051007 22534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:27.052428 22534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:27.064352 22545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:27.064352 22541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:27.064383 22540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:27.064837 22534 server_base.cc:1034] running on GCE node
I20260810 01:47:27.065418 22534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:27.066327 22534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:27.067548 22534 hybrid_clock.cc:648] HybridClock initialized: now 1786326447067537 us; error 34 us; skew 500 ppm
Aug 10 01:47:27 dist-test-slave-kvfs krb5kdc[22123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326447, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:27.071465 22534 init.cc:377] Logged in from keytab as kudu/127.9.77.130@KRBTEST.COM (short username kudu)
I20260810 01:47:27.072960 22534 webserver.cc:460] Webserver started at http://127.9.77.130:36543/ using document root <none> and password file <none>
I20260810 01:47:27.073802 22534 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:27.073916 22534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:27.074244 22534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:27.077203 22534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/data/instance:
uuid: "552ec8a0a7e34aa1a7c631781f2ec7e6"
format_stamp: "Formatted at 2026-08-10 01:47:27 on dist-test-slave-kvfs"
server_key: "dd2ada23fa155edd09e193520bf53624"
server_key_iv: "0ac2be8bc51bc41b01c52393d24045b0"
server_key_version: "encryptionkey@0"
I20260810 01:47:27.077979 22534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance:
uuid: "552ec8a0a7e34aa1a7c631781f2ec7e6"
format_stamp: "Formatted at 2026-08-10 01:47:27 on dist-test-slave-kvfs"
server_key: "dd2ada23fa155edd09e193520bf53624"
server_key_iv: "0ac2be8bc51bc41b01c52393d24045b0"
server_key_version: "encryptionkey@0"
I20260810 01:47:27.083098 22534 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 01:47:27.086266 22555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:27.087788 22534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 01:47:27.087973 22534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "552ec8a0a7e34aa1a7c631781f2ec7e6"
format_stamp: "Formatted at 2026-08-10 01:47:27 on dist-test-slave-kvfs"
server_key: "dd2ada23fa155edd09e193520bf53624"
server_key_iv: "0ac2be8bc51bc41b01c52393d24045b0"
server_key_version: "encryptionkey@0"
I20260810 01:47:27.088192 22534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:27.120850 22534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:27.125119 22534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:27.125454 22534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:27.126263 22534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:27.127399 22534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:27.127475 22534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:27.127559 22534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:27.127604 22534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:27.138756 22534 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:38027
I20260810 01:47:27.138779 22740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:38027 every 8 connection(s)
I20260810 01:47:27.139925 22534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:47:27.140941 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 22534
I20260810 01:47:27.141099 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance
I20260810 01:47:27.141531 9526 external_mini_cluster.cc:1428] Setting key f700f009d03f74f723cbb97821df1c0e
Aug 10 01:47:27 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326447, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.77.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.131@KRBTEST.COM" created.
I20260810 01:47:27.156618 22252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:27.142426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:33997 (local address 127.9.77.190:45897)
0810 01:47:27.142608 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:27.142614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:27.144783 (+ 2169us) server_negotiation.cc:407] Connection header received
0810 01:47:27.145836 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:27.145839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:27.145905 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:27.146019 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:27.147423 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:27.148094 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:27.149378 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:27.149685 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:27.149810 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:27.152672 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:27.152676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:27.152679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:27.152715 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:27.154302 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:27.154859 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:27.154863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:27.154865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:27.154958 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:27.155316 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:27.155319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:27.155321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:27.155536 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:27.155653 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:27.156254 (+ 601us) server_negotiation.cc:299] Negotiation successful
0810 01:47:27.156419 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 01:47:27.157902 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:27.143030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.130:33997)
0810 01:47:27.144614 (+ 1584us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:27.144662 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:27.145599 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:27.146227 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:27.146234 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:27.146666 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:27.147255 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:27.147270 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:27.148229 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:27.148238 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:27.149224 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:27.149233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:27.149859 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:27.149863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:27.150062 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:27.150784 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:27.150802 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:27.152491 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:27.154445 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:27.154451 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:27.154469 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:27.154747 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:27.155059 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:27.155062 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:27.155064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:27.155217 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:27.155651 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:27.155659 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:27.155950 (+ 291us) client_negotiation.cc:769] Sending connection context
0810 01:47:27.156205 (+ 255us) client_negotiation.cc:240] Negotiation successful
0810 01:47:27.156603 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1355,"thread_start_us":158,"threads_started":1}
I20260810 01:47:27.159461 22741 heartbeater.cc:347] Connected to a master server at 127.9.77.190:45897
I20260810 01:47:27.159796 22741 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:27.160586 22741 heartbeater.cc:511] Master 127.9.77.190:45897 requested a full tablet report, sending...
I20260810 01:47:27.162029 22189 ts_manager.cc:194] Registered new tserver with Master: 552ec8a0a7e34aa1a7c631781f2ec7e6 (127.9.77.130:38027)
I20260810 01:47:27.162873 22189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.130@KRBTEST.COM'} at 127.9.77.130:33997
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
I20260810 01:47:27.175624 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:0
--local_ip_for_outbound_sockets=127.9.77.131
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:45897
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:27.321731 22754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:27.322132 22754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:27.322211 22754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:27.327100 22754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:27.327227 22754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:27.327334 22754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:47:27.333847 22754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:45897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:27.335518 22754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:27.336947 22754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:27.348851 22762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:27.348851 22765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:27.348855 22771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:27.349181 22754 server_base.cc:1034] running on GCE node
I20260810 01:47:27.349785 22754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:27.350605 22754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:27.351838 22754 hybrid_clock.cc:648] HybridClock initialized: now 1786326447351797 us; error 53 us; skew 500 ppm
Aug 10 01:47:27 dist-test-slave-kvfs krb5kdc[22123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326447, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:27.355814 22754 init.cc:377] Logged in from keytab as kudu/127.9.77.131@KRBTEST.COM (short username kudu)
I20260810 01:47:27.357018 22754 webserver.cc:460] Webserver started at http://127.9.77.131:35405/ using document root <none> and password file <none>
I20260810 01:47:27.357741 22754 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:27.357854 22754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:27.358129 22754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:27.361078 22754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/data/instance:
uuid: "ca65be78e0e846e0958cdbbc0d7bea2d"
format_stamp: "Formatted at 2026-08-10 01:47:27 on dist-test-slave-kvfs"
server_key: "575f00d85f35b40a04a564d6a5284951"
server_key_iv: "c23e7fc5d1ff6c98b7592fd91fa45b85"
server_key_version: "encryptionkey@0"
I20260810 01:47:27.361795 22754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance:
uuid: "ca65be78e0e846e0958cdbbc0d7bea2d"
format_stamp: "Formatted at 2026-08-10 01:47:27 on dist-test-slave-kvfs"
server_key: "575f00d85f35b40a04a564d6a5284951"
server_key_iv: "c23e7fc5d1ff6c98b7592fd91fa45b85"
server_key_version: "encryptionkey@0"
I20260810 01:47:27.366525 22754 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 01:47:27.369540 22782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:27.371109 22754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 01:47:27.371307 22754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "ca65be78e0e846e0958cdbbc0d7bea2d"
format_stamp: "Formatted at 2026-08-10 01:47:27 on dist-test-slave-kvfs"
server_key: "575f00d85f35b40a04a564d6a5284951"
server_key_iv: "c23e7fc5d1ff6c98b7592fd91fa45b85"
server_key_version: "encryptionkey@0"
I20260810 01:47:27.371455 22754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:27.387490 22754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:27.391776 22754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:27.392082 22754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:27.392976 22754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:27.394198 22754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:27.394273 22754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:27.394358 22754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:27.394413 22754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:27.405612 22754 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:38135
I20260810 01:47:27.405638 22994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:38135 every 8 connection(s)
I20260810 01:47:27.406852 22754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:47:27.416411 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 22754
I20260810 01:47:27.416618 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance
I20260810 01:47:27.416997 9526 external_mini_cluster.cc:1428] Setting key 7d752af2751f9e202e8f4efc8f02637b
Aug 10 01:47:27 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326447, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:47:27.423751 22252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:27.409255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:44669 (local address 127.9.77.190:45897)
0810 01:47:27.409503 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:27.409508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:27.410763 (+ 1255us) server_negotiation.cc:407] Connection header received
0810 01:47:27.411918 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:27.411922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:27.411996 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:27.412159 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:27.413612 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:27.414220 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:27.415507 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:27.415743 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:27.415867 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:27.419601 (+ 3734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:27.419607 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:27.419610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:27.419650 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:27.421251 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:27.421943 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:27.421950 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:27.421953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:27.422052 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:27.422414 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:27.422419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:27.422420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:27.422651 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:27.422780 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:27.423402 (+ 622us) server_negotiation.cc:299] Negotiation successful
0810 01:47:27.423557 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260810 01:47:27.425122 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:27.409747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.131:44669)
0810 01:47:27.410569 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:27.410621 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:27.411634 (+ 1013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:27.412380 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:27.412388 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:27.412836 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:27.413438 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:27.413454 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:27.414359 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:27.414366 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:27.415357 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:27.415366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:27.415930 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:27.415934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:27.416191 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:27.417094 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:27.417124 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:27.419407 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:27.421384 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:27.421392 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:27.421413 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:27.421831 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:27.422151 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:27.422155 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:27.422157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:27.422314 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:27.422793 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:27.422802 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:27.423127 (+ 325us) client_negotiation.cc:769] Sending connection context
0810 01:47:27.423351 (+ 224us) client_negotiation.cc:240] Negotiation successful
0810 01:47:27.423769 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":148,"threads_started":1}
I20260810 01:47:27.426538 22996 heartbeater.cc:347] Connected to a master server at 127.9.77.190:45897
I20260810 01:47:27.426899 22996 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:27.427611 22996 heartbeater.cc:511] Master 127.9.77.190:45897 requested a full tablet report, sending...
I20260810 01:47:27.429203 22189 ts_manager.cc:194] Registered new tserver with Master: ca65be78e0e846e0958cdbbc0d7bea2d (127.9.77.131:38135)
I20260810 01:47:27.429979 22189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.131@KRBTEST.COM'} at 127.9.77.131:44669
I20260810 01:47:27.431772 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.77.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.77.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.77.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.129.keytab.
Entry for principal oryx/127.9.77.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.129.keytab.
WARNING: no policy specified for oryx/127.9.77.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.77.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.77.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.130.keytab.
Entry for principal oryx/127.9.77.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.130.keytab.
WARNING: no policy specified for oryx/127.9.77.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.77.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.77.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.131.keytab.
Entry for principal oryx/127.9.77.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.131.keytab.
I20260810 01:47:27.531744 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 22292
I20260810 01:47:27.549811 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 22534
I20260810 01:47:27.577044 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 22754
I20260810 01:47:27.600196 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 22142
I20260810 01:47:27.614624 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:45897
--webserver_interface=127.9.77.190
--webserver_port=41135
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:45897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:27.753237 23015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:27.753644 23015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:27.753719 23015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:27.758317 23015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:47:27.758422 23015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:27.758461 23015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:27.758495 23015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:47:27.758518 23015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:47:27.769475 23015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:45897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:45897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=41135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:27.771369 23015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:27.773125 23015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:27.787601 23022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:27.787604 23025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:27.788009 23028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:27.788178 23015 server_base.cc:1034] running on GCE node
I20260810 01:47:27.789153 23015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:27.791035 23015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:27.792241 23015 hybrid_clock.cc:648] HybridClock initialized: now 1786326447792204 us; error 53 us; skew 500 ppm
Aug 10 01:47:27 dist-test-slave-kvfs krb5kdc[22123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326447, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:27.797163 23015 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:47:27.801035 23015 webserver.cc:460] Webserver started at http://127.9.77.190:41135/ using document root <none> and password file <none>
I20260810 01:47:27.801762 23015 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:27.801851 23015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:27.807660 23015 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 01:47:27.811301 23039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:27.813469 23015 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260810 01:47:27.813645 23015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "0574e6491e784c879226221e57c30c0c"
format_stamp: "Formatted at 2026-08-10 01:47:26 on dist-test-slave-kvfs"
server_key: "32d23e93fa0f2404fddf78eec3e9df1c"
server_key_iv: "6c289cdb50705b31970314e86906a8f1"
server_key_version: "encryptionkey@0"
I20260810 01:47:27.814700 23015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:27.836601 23015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:27.840138 23015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:27.840332 23015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:27.849318 23015 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:45897
I20260810 01:47:27.849335 23129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:45897 every 8 connection(s)
I20260810 01:47:27.850978 23015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:47:27.855593 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 23015
I20260810 01:47:27.856245 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:37281
--local_ip_for_outbound_sockets=127.9.77.129
--tserver_master_addrs=127.9.77.190:45897
--webserver_port=44029
--webserver_interface=127.9.77.129
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.77.129
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 01:47:27.856871 23130 sys_catalog.cc:263] Verifying existing consensus state
I20260810 01:47:27.858996 23130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c: Bootstrap starting.
I20260810 01:47:27.880306 23130 log.cc:824] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:27.889115 23130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 01:47:27.889582 23130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c: Bootstrap complete.
I20260810 01:47:27.895001 23130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0574e6491e784c879226221e57c30c0c" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45897 } }
I20260810 01:47:27.895514 23130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0574e6491e784c879226221e57c30c0c, State: Initialized, Role: FOLLOWER
I20260810 01:47:27.896217 23130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0574e6491e784c879226221e57c30c0c" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45897 } }
I20260810 01:47:27.896390 23130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:47:27.896481 23130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:47:27.896615 23130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [term 1 FOLLOWER]: Advancing to term 2
I20260810 01:47:27.899009 23130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0574e6491e784c879226221e57c30c0c" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45897 } }
I20260810 01:47:27.899442 23130 leader_election.cc:302] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0574e6491e784c879226221e57c30c0c; no voters:
I20260810 01:47:27.899891 23130 leader_election.cc:288] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 01:47:27.900003 23138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [term 2 FOLLOWER]: Leader election won for term 2
I20260810 01:47:27.900270 23138 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [term 2 LEADER]: Becoming Leader. State: Replica: 0574e6491e784c879226221e57c30c0c, State: Running, Role: LEADER
I20260810 01:47:27.900614 23138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0574e6491e784c879226221e57c30c0c" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45897 } }
I20260810 01:47:27.901464 23130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:47:27.902235 23139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0574e6491e784c879226221e57c30c0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0574e6491e784c879226221e57c30c0c" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45897 } } }
I20260810 01:47:27.902227 23143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0574e6491e784c879226221e57c30c0c. Latest consensus state: current_term: 2 leader_uuid: "0574e6491e784c879226221e57c30c0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0574e6491e784c879226221e57c30c0c" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45897 } } }
I20260810 01:47:27.902417 23143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:27.902415 23139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:27.903159 23148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:47:27.907202 23148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:47:27.908227 23148 catalog_manager.cc:1259] Loaded cluster ID: d3c75ad327914bc38ac97da24f5f6dd7
I20260810 01:47:27.908293 23148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:47:27.911002 23148 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:47:27.911998 23148 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0574e6491e784c879226221e57c30c0c: Loaded TSK: 0
I20260810 01:47:27.913733 23148 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 01:47:27.999329 23132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:27.999665 23132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:27.999713 23132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:28.004416 23132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:28.004505 23132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:28.004601 23132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:47:28.010637 23132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.129.keytab
--principal=oryx/127.9.77.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:37281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=44029
--tserver_master_addrs=127.9.77.190:45897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:28.012190 23132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:28.013561 23132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:28.026947 23178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:28.026978 23180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:28.026978 23171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:28.027582 23132 server_base.cc:1034] running on GCE node
I20260810 01:47:28.028156 23132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:28.029017 23132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:28.030210 23132 hybrid_clock.cc:648] HybridClock initialized: now 1786326448030175 us; error 52 us; skew 500 ppm
Aug 10 01:47:28 dist-test-slave-kvfs krb5kdc[22123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326448, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:28.033725 23132 init.cc:377] Logged in from keytab as oryx/127.9.77.129@KRBTEST.COM (short username oryx)
I20260810 01:47:28.034840 23132 webserver.cc:460] Webserver started at http://127.9.77.129:44029/ using document root <none> and password file <none>
I20260810 01:47:28.035564 23132 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:28.035699 23132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:28.040894 23132 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:47:28.043727 23190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:28.045342 23132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:47:28.045529 23132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "1edc4ceb962f467dbbe10a604a8eb62e"
format_stamp: "Formatted at 2026-08-10 01:47:26 on dist-test-slave-kvfs"
server_key: "dee5d24c0862dfb257abdf007e2499aa"
server_key_iv: "fd94970d1938a5e3f0f9f8acc8033456"
server_key_version: "encryptionkey@0"
I20260810 01:47:28.046300 23132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:28.061818 23132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:28.065358 23132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:28.065599 23132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:28.066354 23132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:28.067394 23132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:28.067490 23132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:28.067595 23132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:28.067647 23132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:28.078423 23132 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:37281
I20260810 01:47:28.078450 23366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:37281 every 8 connection(s)
I20260810 01:47:28.079643 23132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:47:28.087123 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 23132
I20260810 01:47:28.087705 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:38027
--local_ip_for_outbound_sockets=127.9.77.130
--tserver_master_addrs=127.9.77.190:45897
--webserver_port=36543
--webserver_interface=127.9.77.130
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.77.130
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 01:47:28 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:28 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
W20260810 01:47:28.095451 23369 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:28.082253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:54065 (local address 127.9.77.190:45897)
0810 01:47:28.082814 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:28.082833 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:28.083318 (+ 485us) server_negotiation.cc:407] Connection header received
0810 01:47:28.084647 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:28.084659 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:28.085097 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:28.085429 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:28.086671 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.088001 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.089382 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.089974 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.090185 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:28.094508 (+ 4323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.129:54065: BlockingRecv error: recv got EOF from 127.9.77.129:54065 (error 108)
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":158,"threads_started":1}
I20260810 01:47:28.095470 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:28.082313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.129:54065)
0810 01:47:28.083125 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:28.083183 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:28.084206 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:28.085524 (+ 1318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:28.085531 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:28.085952 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:28.086494 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:28.086510 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.088211 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.088220 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:28.089230 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:28.089241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.090176 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.090180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:28.090392 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:28.091147 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:28.091167 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:28.094232 (+ 3065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":163,"threads_started":1}
W20260810 01:47:28.096400 23367 heartbeater.cc:644] Failed to heartbeat to 127.9.77.190:45897 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.77.190:45897: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Aug 10 01:47:28 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:28 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:28.102846 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:28.097020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.129:45495)
0810 01:47:28.097167 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:28.097192 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:28.097286 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:28.097556 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:28.097559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:28.097742 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:28.097846 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:28.097852 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.098636 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.098642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:28.099209 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:28.099217 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.099694 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.099697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:28.099802 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:28.100452 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:28.100464 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:28.102675 (+ 2211us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260810 01:47:28.102949 23369 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:28.097129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:45495 (local address 127.9.77.190:45897)
0810 01:47:28.097279 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:28.097283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:28.097299 (+ 16us) server_negotiation.cc:407] Connection header received
0810 01:47:28.097386 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:28.097389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:28.097443 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:28.097541 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:28.097976 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.098521 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.099337 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.099571 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.099687 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:28.102817 (+ 3130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.129:45495: BlockingRecv error: recv got EOF from 127.9.77.129:45495 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 01:47:28 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:28 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:28.109126 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:28.103693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.129:48405)
0810 01:47:28.103794 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:28.103810 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:28.103886 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:28.104136 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:28.104138 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:28.104265 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:28.104343 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:28.104348 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.105374 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.105377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:28.105902 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:28.105910 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.106474 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.106476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:28.106536 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:28.107074 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:28.107086 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:28.109013 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 01:47:28.109390 23367 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 01:47:28.109627 23369 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:28.103771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:48405 (local address 127.9.77.190:45897)
0810 01:47:28.103898 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:28.103902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:28.103918 (+ 16us) server_negotiation.cc:407] Connection header received
0810 01:47:28.103957 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:28.103960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:28.104001 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:28.104110 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:28.104454 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.105270 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.106013 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.106360 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.106459 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:28.109222 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.129:48405: BlockingRecv error: recv got EOF from 127.9.77.129:48405 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 01:47:28.228109 23371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:28.228446 23371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:28.228494 23371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:28.232631 23371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:28.232692 23371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:28.232752 23371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:47:28.238749 23371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.130.keytab
--principal=oryx/127.9.77.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:38027
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=36543
--tserver_master_addrs=127.9.77.190:45897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:28.240113 23371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:28.241496 23371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:28.254034 23381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:28.254055 23384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:28.254048 23382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:28.254877 23371 server_base.cc:1034] running on GCE node
I20260810 01:47:28.255385 23371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:28.256305 23371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:28.257491 23371 hybrid_clock.cc:648] HybridClock initialized: now 1786326448257477 us; error 31 us; skew 500 ppm
Aug 10 01:47:28 dist-test-slave-kvfs krb5kdc[22123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326448, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:28.264541 23371 init.cc:377] Logged in from keytab as oryx/127.9.77.130@KRBTEST.COM (short username oryx)
I20260810 01:47:28.265738 23371 webserver.cc:460] Webserver started at http://127.9.77.130:36543/ using document root <none> and password file <none>
I20260810 01:47:28.266450 23371 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:28.266536 23371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:28.271795 23371 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 01:47:28.274688 23399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:28.276569 23371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:47:28.276741 23371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "552ec8a0a7e34aa1a7c631781f2ec7e6"
format_stamp: "Formatted at 2026-08-10 01:47:27 on dist-test-slave-kvfs"
server_key: "dd2ada23fa155edd09e193520bf53624"
server_key_iv: "0ac2be8bc51bc41b01c52393d24045b0"
server_key_version: "encryptionkey@0"
I20260810 01:47:28.277516 23371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:28.289428 23371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:28.293304 23371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:28.293555 23371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:28.294340 23371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:28.295442 23371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:28.295497 23371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:28.295545 23371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:28.295607 23371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:28.306675 23371 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:38027
I20260810 01:47:28.306767 23617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:38027 every 8 connection(s)
I20260810 01:47:28.308180 23371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:47:28.308665 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 23371
I20260810 01:47:28.309265 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:38135
--local_ip_for_outbound_sockets=127.9.77.131
--tserver_master_addrs=127.9.77.190:45897
--webserver_port=35405
--webserver_interface=127.9.77.131
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:40963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.77.131
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 01:47:28 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:28 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
W20260810 01:47:28.322906 23369 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:28.310782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:57723 (local address 127.9.77.190:45897)
0810 01:47:28.310975 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:28.310981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:28.312559 (+ 1578us) server_negotiation.cc:407] Connection header received
0810 01:47:28.313667 (+ 1108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:28.313671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:28.313755 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:28.313878 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:28.315358 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.316129 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.317455 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.317713 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.317842 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:28.322753 (+ 4911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.130:57723: BlockingRecv error: recv got EOF from 127.9.77.130:57723 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 01:47:28.324255 23621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:28.311373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.130:57723)
0810 01:47:28.312366 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:28.312421 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:28.313417 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:28.314104 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:28.314113 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:28.314561 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:28.315182 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:28.315199 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.316278 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.316282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:28.317314 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:28.317324 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.317912 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.317917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:28.318127 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:28.319006 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:28.319026 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:28.322570 (+ 3544us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":752,"thread_start_us":184,"threads_started":1}
W20260810 01:47:28.324895 23618 heartbeater.cc:644] Failed to heartbeat to 127.9.77.190:45897 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.77.190:45897: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Aug 10 01:47:28 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:28 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:28.331997 23621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:28.325513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.130:52223)
0810 01:47:28.325660 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:28.325677 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:28.325770 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:28.326046 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:28.326049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:28.326200 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:28.326310 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:28.326315 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.327413 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.327415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:28.327996 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:28.328004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.328655 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.328658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:28.328767 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:28.329467 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:28.329489 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:28.331826 (+ 2337us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 01:47:28.332497 23369 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:28.325613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:52223 (local address 127.9.77.190:45897)
0810 01:47:28.325759 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:28.325764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:28.325782 (+ 18us) server_negotiation.cc:407] Connection header received
0810 01:47:28.325846 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:28.325849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:28.325902 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:28.326005 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:28.326427 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.327301 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.328155 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.328522 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.328634 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:28.332116 (+ 3482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.130:52223: BlockingRecv error: recv got EOF from 127.9.77.130:52223 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 10 01:47:28 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:28 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:28.338536 23621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:28.332706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.130:53599)
0810 01:47:28.332851 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:28.332876 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:28.332966 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:28.333364 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:28.333368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:28.333546 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:28.333653 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:28.333660 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.334399 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.334402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:28.335288 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:28.335300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.335726 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.335729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:28.335821 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:28.336403 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:28.336419 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:28.338372 (+ 1953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 01:47:28.338634 23369 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:28.333016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:53599 (local address 127.9.77.190:45897)
0810 01:47:28.333145 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:28.333149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:28.333160 (+ 11us) server_negotiation.cc:407] Connection header received
0810 01:47:28.333196 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:28.333199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:28.333237 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:28.333303 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:28.333767 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.334298 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.335407 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.335614 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.335690 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:28.338500 (+ 2810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.130:53599: BlockingRecv error: recv got EOF from 127.9.77.130:53599 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 01:47:28.339020 23618 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 01:47:28.455186 23620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:28.455576 23620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:28.455657 23620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:28.459900 23620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:28.459996 23620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:28.460109 23620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:47:28.465811 23620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:40963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/krb5kdc/oryx_127.9.77.131.keytab
--principal=oryx/127.9.77.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:38135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=35405
--tserver_master_addrs=127.9.77.190:45897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:28.467235 23620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:28.468686 23620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:28.481526 23630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:28.481526 23636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:28.481587 23633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:28.481616 23620 server_base.cc:1034] running on GCE node
I20260810 01:47:28.482402 23620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:28.483222 23620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:28.484411 23620 hybrid_clock.cc:648] HybridClock initialized: now 1786326448484376 us; error 46 us; skew 500 ppm
Aug 10 01:47:28 dist-test-slave-kvfs krb5kdc[22123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326448, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.77.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:28.487704 23620 init.cc:377] Logged in from keytab as oryx/127.9.77.131@KRBTEST.COM (short username oryx)
I20260810 01:47:28.488821 23620 webserver.cc:460] Webserver started at http://127.9.77.131:35405/ using document root <none> and password file <none>
I20260810 01:47:28.489559 23620 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:28.489648 23620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:28.494431 23620 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 01:47:28.496910 23648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:28.498356 23620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:47:28.498517 23620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "ca65be78e0e846e0958cdbbc0d7bea2d"
format_stamp: "Formatted at 2026-08-10 01:47:27 on dist-test-slave-kvfs"
server_key: "575f00d85f35b40a04a564d6a5284951"
server_key_iv: "c23e7fc5d1ff6c98b7592fd91fa45b85"
server_key_version: "encryptionkey@0"
I20260810 01:47:28.499202 23620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:28.511492 23620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:28.514919 23620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:28.515168 23620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:28.515897 23620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:28.516989 23620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:28.517067 23620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:28.517146 23620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:28.517200 23620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:28.528091 23620 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:38135
I20260810 01:47:28.528162 23843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:38135 every 8 connection(s)
I20260810 01:47:28.529431 23620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:47:28.529902 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 23620
I20260810 01:47:28.541445 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:28.530948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33980 (local address 127.9.77.190:45897)
0810 01:47:28.531124 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:28.531129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:28.531409 (+ 280us) server_negotiation.cc:407] Connection header received
0810 01:47:28.531560 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:28.531563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:28.531624 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:28.531703 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:28.532847 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.533455 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.534497 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.534769 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.534880 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:28.536801 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:28.536808 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:28.536824 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:28.536882 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:28.538890 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:28.539409 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:28.539416 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:28.539423 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:28.539637 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:28.539957 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:28.539959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:28.539961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:28.540411 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:28.540669 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:28.540977 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 01:47:28.541278 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 01:47:28 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:28 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
W20260810 01:47:28.543843 23852 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:28.532512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:38539 (local address 127.9.77.190:45897)
0810 01:47:28.532830 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:28.532835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:28.533766 (+ 931us) server_negotiation.cc:407] Connection header received
0810 01:47:28.534798 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:28.534804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:28.534872 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:28.534980 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:28.536919 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.537737 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.539165 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.539451 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.539561 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:28.543655 (+ 4094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.131:38539: BlockingRecv error: recv got EOF from 127.9.77.131:38539 (error 108)
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":123,"threads_started":1}
I20260810 01:47:28.545012 23853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:28.532696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.131:38539)
0810 01:47:28.533593 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:28.533653 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:28.534585 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:28.535250 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:28.535263 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:28.535867 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:28.536751 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:28.536780 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.537923 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.537928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:28.539032 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:28.539044 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.539613 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.539616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:28.539876 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:28.540579 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:28.540596 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:28.543546 (+ 2950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":153,"threads_started":1}
W20260810 01:47:28.545778 23848 heartbeater.cc:644] Failed to heartbeat to 127.9.77.190:45897 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.77.190:45897: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Aug 10 01:47:28 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:28 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:28.551199 23853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:28.546285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.131:40049)
0810 01:47:28.546411 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:28.546428 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:28.546517 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:28.546717 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:28.546719 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:28.546857 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:28.546965 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:28.546969 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.547681 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.547683 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:28.548322 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:28.548331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.548748 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.548751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:28.548825 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:28.549275 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:28.549286 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:28.551101 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 01:47:28.551405 23852 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:28.546387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:40049 (local address 127.9.77.190:45897)
0810 01:47:28.546508 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:28.546511 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:28.546524 (+ 13us) server_negotiation.cc:407] Connection header received
0810 01:47:28.546582 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:28.546584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:28.546622 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:28.546687 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:28.547061 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.547592 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.548427 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.548652 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.548732 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:28.551264 (+ 2532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.131:40049: BlockingRecv error: recv got EOF from 127.9.77.131:40049 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 10 01:47:28 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:28 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:28.556758 23853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:28.551787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.131:47225)
0810 01:47:28.551909 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:28.551922 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:28.552025 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:28.552272 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:28.552275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:28.552444 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:28.552517 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:28.552522 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.553246 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.553251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:28.553782 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:28.553790 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.554240 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.554244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:28.554333 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:28.554770 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:28.554782 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:28.556613 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 01:47:28.556859 23852 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:28.551868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:47225 (local address 127.9.77.190:45897)
0810 01:47:28.552063 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:28.552070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:28.552086 (+ 16us) server_negotiation.cc:407] Connection header received
0810 01:47:28.552125 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:28.552130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:28.552170 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:28.552256 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:28.552632 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.553134 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.553914 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:28.554131 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:28.554240 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:28.556725 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.131:47225: BlockingRecv error: recv got EOF from 127.9.77.131:47225 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260810 01:47:28.557060 23848 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 01:47:29 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:29 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:29.116856 23854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:29.110213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.129:35529)
0810 01:47:29.110581 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:29.110597 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:29.110713 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:29.111024 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:29.111027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:29.111275 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:29.111417 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:29.111425 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:29.112300 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:29.112304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:29.113044 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:29.113055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:29.113556 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:29.113560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:29.113643 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:29.114338 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:29.114354 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:29.116620 (+ 2266us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":97,"threads_started":1}
W20260810 01:47:29.117020 23855 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:29.110333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:35529 (local address 127.9.77.190:45897)
0810 01:47:29.110649 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:29.110653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:29.110668 (+ 15us) server_negotiation.cc:407] Connection header received
0810 01:47:29.110808 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:29.110810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:29.110861 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:29.110945 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:29.111542 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:29.112171 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:29.113164 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:29.113395 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:29.113491 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:29.116818 (+ 3327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.129:35529: BlockingRecv error: recv got EOF from 127.9.77.129:35529 (error 108)
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":124,"threads_started":1}
Aug 10 01:47:29 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:29 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:29.346120 23856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:29.339852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.130:37825)
0810 01:47:29.340204 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:29.340233 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:29.340358 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:29.340669 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:29.340673 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:29.340920 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:29.341055 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:29.341062 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:29.341866 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:29.341869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:29.342502 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:29.342511 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:29.342965 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:29.342969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:29.343111 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:29.343847 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:29.343864 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:29.345972 (+ 2108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":119,"threads_started":1}
W20260810 01:47:29.346323 23855 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:29.339964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:37825 (local address 127.9.77.190:45897)
0810 01:47:29.340157 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:29.340161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:29.340314 (+ 153us) server_negotiation.cc:407] Connection header received
0810 01:47:29.340448 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:29.340450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:29.340503 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:29.340585 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:29.341171 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:29.341752 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:29.342610 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:29.342839 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:29.342938 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:29.346141 (+ 3203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.130:37825: BlockingRecv error: recv got EOF from 127.9.77.130:37825 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 10 01:47:29 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:29 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:29.564620 23857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:29.557930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.131:59143)
0810 01:47:29.558387 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:29.558405 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:29.558581 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:29.558990 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:29.558993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:29.559313 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:29.559490 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:29.559502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:29.560455 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:29.560458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:29.561085 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:29.561093 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:29.561580 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:29.561582 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:29.561651 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:29.562237 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:29.562249 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:29.564467 (+ 2218us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":114,"threads_started":1}
W20260810 01:47:29.564775 23855 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:29.558013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:59143 (local address 127.9.77.190:45897)
0810 01:47:29.558226 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:29.558233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:29.558535 (+ 302us) server_negotiation.cc:407] Connection header received
0810 01:47:29.558698 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:29.558703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:29.558797 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:29.558921 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:29.559628 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:29.560349 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:29.561233 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:29.561472 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:29.561559 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:29.564627 (+ 3068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.131:59143: BlockingRecv error: recv got EOF from 127.9.77.131:59143 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 10 01:47:30 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:30 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:30.124729 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:30.118029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.129:58423)
0810 01:47:30.118373 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:30.118389 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:30.118527 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:30.118846 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:30.118848 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:30.119085 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:30.119212 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:30.119218 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:30.120156 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:30.120159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:30.120777 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:30.120789 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:30.121356 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:30.121359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:30.121444 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:30.122090 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:30.122103 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:30.124541 (+ 2438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":122,"threads_started":1}
W20260810 01:47:30.124864 23860 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:30.118073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:58423 (local address 127.9.77.190:45897)
0810 01:47:30.118415 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:30.118418 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:30.118476 (+ 58us) server_negotiation.cc:407] Connection header received
0810 01:47:30.118629 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:30.118632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:30.118686 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:30.118804 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:30.119348 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:30.120006 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:30.120971 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:30.121237 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:30.121330 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:30.124689 (+ 3359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.129:58423: BlockingRecv error: recv got EOF from 127.9.77.129:58423 (error 108)
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":155,"threads_started":1}
Aug 10 01:47:30 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:30 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:30.354023 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:30.347219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.130:51711)
0810 01:47:30.347637 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:30.347652 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:30.347829 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:30.348201 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:30.348204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:30.348532 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:30.348721 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:30.348730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:30.349664 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:30.349667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:30.350270 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:30.350278 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:30.350806 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:30.350808 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:30.350875 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:30.351494 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:30.351506 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:30.353842 (+ 2336us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":199,"threads_started":1}
W20260810 01:47:30.354174 23860 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:30.347296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:51711 (local address 127.9.77.190:45897)
0810 01:47:30.347514 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:30.347520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:30.347750 (+ 230us) server_negotiation.cc:407] Connection header received
0810 01:47:30.347944 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:30.347948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:30.348044 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:30.348133 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:30.348858 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:30.349557 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:30.350398 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:30.350681 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:30.350815 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:30.354031 (+ 3216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.130:51711: BlockingRecv error: recv got EOF from 127.9.77.130:51711 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 01:47:30 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:30 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:30.572469 23864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:30.565675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.131:44723)
0810 01:47:30.566124 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:30.566140 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:30.566302 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:30.566622 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:30.566626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:30.566897 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:30.567042 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:30.567049 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:30.567920 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:30.567925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:30.568584 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:30.568593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:30.569103 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:30.569107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:30.569193 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:30.569854 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:30.569867 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:30.572268 (+ 2401us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":181,"threads_started":1}
W20260810 01:47:30.572760 23860 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:30.565875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:44723 (local address 127.9.77.190:45897)
0810 01:47:30.566065 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:30.566070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:30.566224 (+ 154us) server_negotiation.cc:407] Connection header received
0810 01:47:30.566417 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:30.566420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:30.566491 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:30.566590 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:30.567178 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:30.567799 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:30.568724 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:30.568975 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:30.569095 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:30.572460 (+ 3365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.131:44723: BlockingRecv error: recv got EOF from 127.9.77.131:44723 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 10 01:47:31 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:31 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:31.132274 23867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:31.125664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.129:58897)
0810 01:47:31.126005 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:31.126019 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:31.126141 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:31.126532 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:31.126535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:31.126784 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:31.126928 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:31.126935 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:31.127854 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:31.127857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:31.128471 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:31.128479 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:31.128987 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:31.128989 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:31.129060 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:31.129748 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:31.129762 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:31.132111 (+ 2349us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":123,"threads_started":1}
W20260810 01:47:31.132452 23869 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:31.125743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:58897 (local address 127.9.77.190:45897)
0810 01:47:31.126044 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:31.126048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:31.126082 (+ 34us) server_negotiation.cc:407] Connection header received
0810 01:47:31.126288 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:31.126291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:31.126352 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:31.126452 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:31.127067 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:31.127721 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:31.128592 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:31.128871 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:31.128970 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:31.132254 (+ 3284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.129:58897: BlockingRecv error: recv got EOF from 127.9.77.129:58897 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":102,"threads_started":1}
Aug 10 01:47:31 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:31 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:31.361649 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:31.355015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.130:40859)
0810 01:47:31.355392 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:31.355407 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:31.355576 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:31.355893 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:31.355896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:31.356201 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:31.356329 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:31.356335 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:31.357191 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:31.357196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:31.357821 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:31.357830 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:31.358326 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:31.358329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:31.358426 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:31.359111 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:31.359125 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:31.361425 (+ 2300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":123,"threads_started":1}
W20260810 01:47:31.361697 23869 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:31.355101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:40859 (local address 127.9.77.190:45897)
0810 01:47:31.355309 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:31.355315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:31.355502 (+ 187us) server_negotiation.cc:407] Connection header received
0810 01:47:31.355694 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:31.355700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:31.355763 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:31.355860 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:31.356457 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:31.357069 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:31.357958 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:31.358202 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:31.358320 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:31.361573 (+ 3253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.130:40859: BlockingRecv error: recv got EOF from 127.9.77.130:40859 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 01:47:31 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:31 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:31.580164 23872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:31.573627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.131:43647)
0810 01:47:31.573980 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:31.573995 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:31.574131 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:31.574506 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:31.574509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:31.574767 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:31.574890 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:31.574896 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:31.575883 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:31.575886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:31.576566 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:31.576576 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:31.577062 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:31.577066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:31.577164 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:31.577778 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:31.577792 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:31.579917 (+ 2125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":124,"threads_started":1}
W20260810 01:47:31.580204 23869 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:31.573792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:43647 (local address 127.9.77.190:45897)
0810 01:47:31.573978 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:31.573982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:31.574081 (+ 99us) server_negotiation.cc:407] Connection header received
0810 01:47:31.574272 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:31.574290 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:31.574365 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:31.574466 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:31.575057 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:31.575754 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:31.576697 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:31.576939 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:31.577056 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:31.580086 (+ 3030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.131:43647: BlockingRecv error: recv got EOF from 127.9.77.131:43647 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 10 01:47:32 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:32 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:32.139936 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:32.133331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.129:33005)
0810 01:47:32.133707 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:32.133722 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:32.133856 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:32.134196 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:32.134199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:32.134437 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:32.134564 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:32.134570 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:32.135417 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:32.135420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:32.136015 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:32.136091 (+ 76us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:32.136572 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:32.136575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:32.136653 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:32.137335 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:32.137360 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:32.139728 (+ 2368us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":141,"threads_started":1}
W20260810 01:47:32.140110 23876 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:32.133358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:33005 (local address 127.9.77.190:45897)
0810 01:47:32.133754 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:32.133758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:32.133774 (+ 16us) server_negotiation.cc:407] Connection header received
0810 01:47:32.133955 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:32.133957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:32.134012 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:32.134092 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:32.134680 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:32.135302 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:32.136207 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:32.136461 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:32.136545 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:32.139923 (+ 3378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.129:33005: BlockingRecv error: recv got EOF from 127.9.77.129:33005 (error 108)
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":117,"threads_started":1}
Aug 10 01:47:32 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:32 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:32.369801 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:32.362886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.130:55397)
0810 01:47:32.363316 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:32.363338 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:32.363543 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:32.363893 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:32.363897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:32.364229 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:32.364368 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:32.364375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:32.365340 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:32.365346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:32.366022 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:32.366032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:32.366571 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:32.366575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:32.366677 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:32.367368 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:32.367381 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:32.369590 (+ 2209us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":142,"threads_started":1}
W20260810 01:47:32.369903 23876 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:32.363072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:55397 (local address 127.9.77.190:45897)
0810 01:47:32.363299 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:32.363306 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:32.363448 (+ 142us) server_negotiation.cc:407] Connection header received
0810 01:47:32.363674 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:32.363678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:32.363750 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:32.363855 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:32.364522 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:32.365206 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:32.366160 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:32.366434 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:32.366566 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:32.369744 (+ 3178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.130:55397: BlockingRecv error: recv got EOF from 127.9.77.130:55397 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 10 01:47:32 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:32 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:32.587966 23878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:32.581229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.131:45697)
0810 01:47:32.581659 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:32.581674 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:32.581837 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:32.582141 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:32.582143 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:32.582394 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:32.582520 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:32.582526 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:32.583480 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:32.583484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:32.584193 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:32.584204 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:32.584690 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:32.584695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:32.584786 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:32.585499 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:32.585512 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:32.587823 (+ 2311us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":137,"threads_started":1}
W20260810 01:47:32.588222 23876 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:32.581366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:45697 (local address 127.9.77.190:45897)
0810 01:47:32.581593 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:32.581598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:32.581767 (+ 169us) server_negotiation.cc:407] Connection header received
0810 01:47:32.581952 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:32.581954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:32.582020 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:32.582106 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:32.582658 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:32.583346 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:32.584316 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:32.584548 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:32.584629 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:32.588001 (+ 3372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.131:45697: BlockingRecv error: recv got EOF from 127.9.77.131:45697 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 01:47:33 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:33 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:33.147861 23880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:33.141045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.129:48931)
0810 01:47:33.141440 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:33.141456 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:33.141571 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:33.141868 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:33.141870 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:33.142112 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:33.142237 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:33.142243 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:33.143241 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:33.143245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:33.143887 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:33.143896 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:33.144629 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:33.144633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:33.144718 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:33.145466 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:33.145480 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:33.147704 (+ 2224us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":178,"threads_started":1}
W20260810 01:47:33.148164 23881 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:33.141161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:48931 (local address 127.9.77.190:45897)
0810 01:47:33.141466 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:33.141470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:33.141534 (+ 64us) server_negotiation.cc:407] Connection header received
0810 01:47:33.141662 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:33.141664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:33.141722 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:33.141815 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:33.142410 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:33.143072 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:33.144203 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:33.144475 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:33.144578 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:33.147964 (+ 3386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.129:48931: BlockingRecv error: recv got EOF from 127.9.77.129:48931 (error 108)
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":115,"threads_started":1}
Aug 10 01:47:33 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:33 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:33.382510 23883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:33.371285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.130:59475)
0810 01:47:33.371883 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:33.371903 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:33.372096 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:33.372470 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:33.372473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:33.372816 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:33.372989 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:33.372999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:33.374130 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:33.374133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:33.374751 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:33.374759 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:33.375252 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:33.375254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:33.375332 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:33.379608 (+ 4276us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:33.379626 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:33.382333 (+ 2707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":184,"threads_started":1}
W20260810 01:47:33.382771 23881 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:33.371595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:59475 (local address 127.9.77.190:45897)
0810 01:47:33.371778 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:33.371783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:33.372111 (+ 328us) server_negotiation.cc:407] Connection header received
0810 01:47:33.372231 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:33.372234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:33.372314 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:33.372413 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:33.373163 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:33.374009 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:33.374876 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:33.375139 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:33.375229 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:33.382599 (+ 7370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.130:59475: BlockingRecv error: recv got EOF from 127.9.77.130:59475 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 01:47:33 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:33 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:33.595693 23887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:33.589224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.131:35439)
0810 01:47:33.589626 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:33.589642 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:33.589772 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:33.590051 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:33.590054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:33.590299 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:33.590437 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:33.590443 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:33.591301 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:33.591304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:33.592080 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:33.592092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:33.592694 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:33.592698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:33.592787 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:33.593477 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:33.593491 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:33.595547 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":177,"threads_started":1}
W20260810 01:47:33.595856 23881 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:33.589302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:35439 (local address 127.9.77.190:45897)
0810 01:47:33.589503 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:33.589508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:33.589734 (+ 226us) server_negotiation.cc:407] Connection header received
0810 01:47:33.589861 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:33.589864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:33.589926 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:33.590008 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:33.590551 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:33.591156 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:33.592216 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:33.592564 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:33.592690 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:33.595713 (+ 3023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.131:35439: BlockingRecv error: recv got EOF from 127.9.77.131:35439 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 10 01:47:34 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:34 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:34.156126 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:34.149144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.129:33239)
0810 01:47:34.149586 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:34.149602 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:34.149724 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:34.150035 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:34.150037 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:34.150347 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:34.150471 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:34.150500 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:34.151769 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:34.151773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:34.152627 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:34.152638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:34.153111 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:34.153114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:34.153189 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:34.153828 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:34.153841 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:34.155957 (+ 2116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":135,"threads_started":1}
W20260810 01:47:34.156369 23889 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:34.149237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:33239 (local address 127.9.77.190:45897)
0810 01:47:34.149653 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:34.149658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:34.149674 (+ 16us) server_negotiation.cc:407] Connection header received
0810 01:47:34.149826 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:34.149828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:34.149886 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:34.149997 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:34.150638 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:34.151534 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:34.152743 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:34.152994 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:34.153083 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:34.156189 (+ 3106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.129:33239: BlockingRecv error: recv got EOF from 127.9.77.129:33239 (error 108)
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":209,"threads_started":1}
Aug 10 01:47:34 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:34 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:34.390247 23890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:34.383522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.130:38469)
0810 01:47:34.383887 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:34.383904 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:34.384092 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:34.384423 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:34.384427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:34.384707 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:34.384829 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:34.384835 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:34.385703 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:34.385707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:34.386322 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:34.386331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:34.386857 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:34.386860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:34.386955 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:34.387625 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:34.387640 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:34.390091 (+ 2451us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":156,"threads_started":1}
W20260810 01:47:34.390609 23889 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:34.383682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:38469 (local address 127.9.77.190:45897)
0810 01:47:34.383882 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:34.383886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:34.384006 (+ 120us) server_negotiation.cc:407] Connection header received
0810 01:47:34.384216 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:34.384220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:34.384290 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:34.384383 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:34.384960 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:34.385582 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:34.386449 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:34.386701 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:34.386838 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:34.390287 (+ 3449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.130:38469: BlockingRecv error: recv got EOF from 127.9.77.130:38469 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 10 01:47:34 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:34 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:34.604080 23891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:34.596989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.131:52531)
0810 01:47:34.597440 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:34.597458 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:34.597628 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:34.597998 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:34.598003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:34.598346 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:34.598549 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:34.598561 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:34.599550 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:34.599553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:34.600190 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:34.600199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:34.600651 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:34.600653 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:34.600722 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:34.601320 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:34.601333 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:34.603872 (+ 2539us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":180,"threads_started":1}
W20260810 01:47:34.604218 23889 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:34.597092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:52531 (local address 127.9.77.190:45897)
0810 01:47:34.597287 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:34.597293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:34.597580 (+ 287us) server_negotiation.cc:407] Connection header received
0810 01:47:34.597755 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:34.597762 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:34.597833 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:34.597920 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:34.598727 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:34.599437 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:34.600298 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:34.600548 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:34.600633 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:34.604059 (+ 3426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.131:52531: BlockingRecv error: recv got EOF from 127.9.77.131:52531 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 01:47:35 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:35 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:35.163712 23892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:35.157080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.129:48251)
0810 01:47:35.157537 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:35.157554 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:35.157687 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:35.157984 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:35.157986 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:35.158234 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:35.158362 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:35.158368 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:35.159359 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:35.159363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:35.159986 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:35.159996 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:35.160557 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:35.160562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:35.160665 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:35.161303 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:35.161315 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:35.163493 (+ 2178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":118,"threads_started":1}
W20260810 01:47:35.163794 23893 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:35.157278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:48251 (local address 127.9.77.190:45897)
0810 01:47:35.157570 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:35.157574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:35.157639 (+ 65us) server_negotiation.cc:407] Connection header received
0810 01:47:35.157778 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:35.157780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:35.157833 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:35.157940 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:35.158467 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:35.159223 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:35.160163 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:35.160425 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:35.160544 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:35.163650 (+ 3106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.129:48251: BlockingRecv error: recv got EOF from 127.9.77.129:48251 (error 108)
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":81,"threads_started":1}
Aug 10 01:47:35 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:35 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:35.398221 23895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:35.391204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.130:42399)
0810 01:47:35.391648 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:35.391665 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:35.391794 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:35.392171 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:35.392174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:35.392425 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:35.392565 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:35.392571 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:35.393460 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:35.393465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:35.394135 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:35.394145 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:35.394623 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:35.394626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:35.394703 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:35.395382 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:35.395396 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:35.398026 (+ 2630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":178,"threads_started":1}
W20260810 01:47:35.398674 23893 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:35.391310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:42399 (local address 127.9.77.190:45897)
0810 01:47:35.391532 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:35.391537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:35.391752 (+ 215us) server_negotiation.cc:407] Connection header received
0810 01:47:35.391884 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:35.391887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:35.391964 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:35.392137 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:35.392682 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:35.393318 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:35.394253 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:35.394507 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:35.394593 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:35.398238 (+ 3645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.130:42399: BlockingRecv error: recv got EOF from 127.9.77.130:42399 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 10 01:47:35 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:35 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:35.611930 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:35.605156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.131:35517)
0810 01:47:35.605603 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:35.605619 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:35.605762 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:35.606036 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:35.606038 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:35.606286 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:35.606413 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:35.606418 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:35.607303 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:35.607307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:35.608056 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:35.608068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:35.608597 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:35.608601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:35.608683 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:35.609327 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:35.609341 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:35.611704 (+ 2363us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":143,"threads_started":1}
W20260810 01:47:35.611981 23893 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:35.605214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:35517 (local address 127.9.77.190:45897)
0810 01:47:35.605417 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:35.605421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:35.605711 (+ 290us) server_negotiation.cc:407] Connection header received
0810 01:47:35.605855 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:35.605858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:35.605918 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:35.605995 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:35.606562 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:35.607164 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:35.608183 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:35.608446 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:35.608550 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:35.611861 (+ 3311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.131:35517: BlockingRecv error: recv got EOF from 127.9.77.131:35517 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 10 01:47:36 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:36 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:36.171592 23897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:36.164753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.129:56235)
0810 01:47:36.165124 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:36.165145 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:36.165287 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:36.165582 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:36.165585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:36.165830 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:36.165965 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:36.165971 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:36.166995 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:36.166999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:36.167775 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:36.167787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:36.168401 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:36.168405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:36.168518 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:36.169259 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:36.169273 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:36.171437 (+ 2164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":104,"threads_started":1}
W20260810 01:47:36.171810 23898 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:36.164992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:56235 (local address 127.9.77.190:45897)
0810 01:47:36.165301 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:36.165305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:36.165343 (+ 38us) server_negotiation.cc:407] Connection header received
0810 01:47:36.165390 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:36.165392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:36.165450 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:36.165532 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:36.166130 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:36.166851 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:36.167896 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:36.168275 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:36.168397 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:36.171609 (+ 3212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.129:56235: BlockingRecv error: recv got EOF from 127.9.77.129:56235 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":92,"threads_started":1}
Aug 10 01:47:36 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:36 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:36.406567 23899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:36.399325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.130:55703)
0810 01:47:36.399670 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:36.399685 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:36.399827 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:36.400159 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:36.400162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:36.400413 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:36.400544 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:36.400551 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:36.401659 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:36.401663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:36.402410 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:36.402421 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:36.402959 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:36.402964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:36.403092 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:36.403840 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:36.403854 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:36.406410 (+ 2556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":132,"threads_started":1}
W20260810 01:47:36.406762 23898 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:36.399473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:55703 (local address 127.9.77.190:45897)
0810 01:47:36.399641 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:36.399645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:36.399742 (+ 97us) server_negotiation.cc:407] Connection header received
0810 01:47:36.399912 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:36.399914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:36.399967 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:36.400127 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:36.400729 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:36.401514 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:36.402574 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:36.402838 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:36.402923 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:36.406606 (+ 3683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.130:55703: BlockingRecv error: recv got EOF from 127.9.77.130:55703 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 01:47:36 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:36 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:36.619495 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:36.613062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.131:58463)
0810 01:47:36.613391 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:36.613443 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:36.613613 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:36.613991 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:36.613994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:36.614251 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:36.614378 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:36.614384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:36.615229 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:36.615231 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:36.615831 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:36.615838 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:36.616320 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:36.616323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:36.616397 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:36.617022 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:36.617036 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:36.619278 (+ 2242us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":114,"threads_started":1}
W20260810 01:47:36.619580 23898 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:36.613113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:58463 (local address 127.9.77.190:45897)
0810 01:47:36.613304 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:36.613309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:36.613597 (+ 288us) server_negotiation.cc:407] Connection header received
0810 01:47:36.613699 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:36.613704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:36.613807 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:36.613926 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:36.614506 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:36.615126 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:36.615941 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:36.616216 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:36.616302 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:36.619433 (+ 3131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.131:58463: BlockingRecv error: recv got EOF from 127.9.77.131:58463 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 10 01:47:37 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:37 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:37.179670 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:37.172715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.129:42433)
0810 01:47:37.173068 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:37.173104 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:37.173254 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:37.173631 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:37.173633 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:37.173877 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:37.174001 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:37.174007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:37.174907 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:37.174911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:37.175512 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:37.175520 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:37.176176 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:37.176180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:37.176256 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:37.176899 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:37.176911 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:37.179410 (+ 2499us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":121,"threads_started":1}
W20260810 01:47:37.179740 23908 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:37.172777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:42433 (local address 127.9.77.190:45897)
0810 01:47:37.173114 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:37.173119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:37.173188 (+ 69us) server_negotiation.cc:407] Connection header received
0810 01:47:37.173400 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:37.173405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:37.173481 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:37.173601 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:37.174112 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:37.174799 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:37.175638 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:37.175912 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:37.176006 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:37.179577 (+ 3571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.129:42433: BlockingRecv error: recv got EOF from 127.9.77.129:42433 (error 108)
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":132,"threads_started":1}
Aug 10 01:47:37 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:37 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:37.414572 23909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:37.407790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.130:41279)
0810 01:47:37.408222 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:37.408237 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:37.408410 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:37.408731 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:37.408734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:37.409028 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:37.409151 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:37.409158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:37.410027 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:37.410032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:37.410694 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:37.410703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:37.411217 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:37.411220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:37.411315 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:37.411983 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:37.411998 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:37.414328 (+ 2330us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":129,"threads_started":1}
W20260810 01:47:37.414639 23908 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:37.407912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:41279 (local address 127.9.77.190:45897)
0810 01:47:37.408162 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:37.408168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:37.408331 (+ 163us) server_negotiation.cc:407] Connection header received
0810 01:47:37.408530 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:37.408533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:37.408599 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:37.408703 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:37.409292 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:37.409905 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:37.410838 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:37.411090 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:37.411210 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:37.414502 (+ 3292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.130:41279: BlockingRecv error: recv got EOF from 127.9.77.130:41279 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 10 01:47:37 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:37 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:37.627388 23912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:37.620648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.131:46823)
0810 01:47:37.621021 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:37.621036 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:37.621166 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:37.621436 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:37.621438 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:37.621680 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:37.621829 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:37.621835 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:37.622747 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:37.622749 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:37.623464 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:37.623484 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:37.624087 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:37.624091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:37.624192 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:37.624924 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:37.624938 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:37.627212 (+ 2274us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":153,"threads_started":1}
W20260810 01:47:37.627617 23908 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:37.620743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:46823 (local address 127.9.77.190:45897)
0810 01:47:37.620939 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:37.620947 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:37.621144 (+ 197us) server_negotiation.cc:407] Connection header received
0810 01:47:37.621247 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:37.621249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:37.621311 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:37.621392 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:37.621981 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:37.622639 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:37.623581 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:37.623933 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:37.624059 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:37.627406 (+ 3347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.131:46823: BlockingRecv error: recv got EOF from 127.9.77.131:46823 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 01:47:38 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:38 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:38.188181 23913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:38.180974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.129:50711)
0810 01:47:38.181332 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:38.181348 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:38.181490 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:38.181887 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:38.181890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:38.182138 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:38.182271 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:38.182278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:38.183286 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:38.183289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:38.183912 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:38.183921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:38.184407 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:38.184410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:38.184499 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:38.185232 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:38.185247 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:38.187944 (+ 2697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":130,"threads_started":1}
W20260810 01:47:38.188537 23914 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:38.181056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:50711 (local address 127.9.77.190:45897)
0810 01:47:38.181379 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:38.181384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:38.181447 (+ 63us) server_negotiation.cc:407] Connection header received
0810 01:47:38.181622 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:38.181625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:38.181682 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:38.181799 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:38.182434 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:38.183164 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:38.184053 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:38.184298 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:38.184381 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:38.188145 (+ 3764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.129:50711: BlockingRecv error: recv got EOF from 127.9.77.129:50711 (error 108)
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":139,"threads_started":1}
Aug 10 01:47:38 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:38 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:38.426374 23916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:38.415673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.130:53557)
0810 01:47:38.416076 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:38.416091 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:38.416213 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:38.416496 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:38.416499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:38.416768 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:38.416901 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:38.416908 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:38.417754 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:38.417758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:38.418390 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:38.418400 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:38.418916 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:38.418921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:38.419023 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:38.423837 (+ 4814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:38.423852 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:38.426169 (+ 2317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":153,"threads_started":1}
W20260810 01:47:38.426492 23914 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:38.415803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:53557 (local address 127.9.77.190:45897)
0810 01:47:38.415991 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:38.415995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:38.416174 (+ 179us) server_negotiation.cc:407] Connection header received
0810 01:47:38.416312 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:38.416315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:38.416371 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:38.416453 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:38.417012 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:38.417612 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:38.418505 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:38.418748 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:38.418857 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:38.426352 (+ 7495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.130:53557: BlockingRecv error: recv got EOF from 127.9.77.130:53557 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 10 01:47:38 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:38 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:38.635035 23917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:38.628467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.131:41367)
0810 01:47:38.628839 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:38.628858 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:38.628986 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:38.629317 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:38.629320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:38.629559 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:38.629683 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:38.629689 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:38.630529 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:38.630532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:38.631154 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:38.631162 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:38.631631 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:38.631635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:38.631710 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:38.632469 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:38.632483 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:38.634814 (+ 2331us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":97,"threads_started":1}
W20260810 01:47:38.635128 23914 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:38.628670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:41367 (local address 127.9.77.190:45897)
0810 01:47:38.628861 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:38.628865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:38.628986 (+ 121us) server_negotiation.cc:407] Connection header received
0810 01:47:38.629087 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:38.629089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:38.629151 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:38.629229 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:38.629809 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:38.630412 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:38.631262 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:38.631500 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:38.631578 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:38.634980 (+ 3402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.131:41367: BlockingRecv error: recv got EOF from 127.9.77.131:41367 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 10 01:47:39 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:39 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:39.196188 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:39.189361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.129:42701)
0810 01:47:39.189879 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:39.189895 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:39.190055 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:39.190429 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:39.190433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:39.190774 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:39.190913 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:39.190920 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:39.191784 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:39.191788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:39.192508 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:39.192519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:39.193067 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:39.193069 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:39.193142 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:39.193780 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:39.193794 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:39.195964 (+ 2170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":128,"threads_started":1}
W20260810 01:47:39.196395 23919 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:39.189457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:42701 (local address 127.9.77.190:45897)
0810 01:47:39.189851 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:39.189858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:39.189986 (+ 128us) server_negotiation.cc:407] Connection header received
0810 01:47:39.190190 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:39.190194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:39.190264 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:39.190380 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:39.191038 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:39.191672 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:39.192683 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:39.192951 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:39.193046 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:39.196178 (+ 3132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.129:42701: BlockingRecv error: recv got EOF from 127.9.77.129:42701 (error 108)
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":117,"threads_started":1}
Aug 10 01:47:39 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:39 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:39.433488 23921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:39.427358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.130:34275)
0810 01:47:39.427696 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:39.427711 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:39.427837 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:39.428175 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:39.428177 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:39.428426 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:39.428566 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:39.428573 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:39.429384 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:39.429387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:39.429994 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:39.430003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:39.430463 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:39.430466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:39.430541 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:39.431136 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:39.431149 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:39.433327 (+ 2178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":143,"threads_started":1}
W20260810 01:47:39.433679 23919 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:39.427470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:34275 (local address 127.9.77.190:45897)
0810 01:47:39.427663 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:39.427668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:39.427771 (+ 103us) server_negotiation.cc:407] Connection header received
0810 01:47:39.427923 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:39.427927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:39.427985 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:39.428164 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:39.428675 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:39.429287 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:39.430113 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:39.430345 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:39.430445 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:39.433496 (+ 3051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.130:34275: BlockingRecv error: recv got EOF from 127.9.77.130:34275 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 10 01:47:39 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:39 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:39.642979 23926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:39.636076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.131:42131)
0810 01:47:39.636461 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:39.636477 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:39.636605 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:39.636947 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:39.636950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:39.637193 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:39.637318 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:39.637323 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:39.638194 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:39.638197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:39.638817 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:39.638826 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:39.639448 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:39.639455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:39.639571 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:39.640427 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:39.640445 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:39.642761 (+ 2316us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":171,"threads_started":1}
W20260810 01:47:39.643049 23919 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:39.636166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:42131 (local address 127.9.77.190:45897)
0810 01:47:39.636376 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:39.636381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:39.636566 (+ 185us) server_negotiation.cc:407] Connection header received
0810 01:47:39.636706 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:39.636709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:39.636771 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:39.636850 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:39.637439 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:39.638063 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:39.638941 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:39.639267 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:39.639395 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:39.642909 (+ 3514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.131:42131: BlockingRecv error: recv got EOF from 127.9.77.131:42131 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 10 01:47:40 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:40 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:40.204092 23927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:40.197246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.129:35535)
0810 01:47:40.197613 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:40.197628 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:40.197751 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:40.198021 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:40.198023 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:40.198282 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:40.198411 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:40.198418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:40.199287 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:40.199291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:40.199900 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:40.199909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:40.200422 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:40.200426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:40.200505 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:40.201127 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:40.201141 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:40.203892 (+ 2751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":133,"threads_started":1}
W20260810 01:47:40.204298 23930 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:40.197378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:35535 (local address 127.9.77.190:45897)
0810 01:47:40.197664 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:40.197668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:40.197686 (+ 18us) server_negotiation.cc:407] Connection header received
0810 01:47:40.197831 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:40.197833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:40.197885 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:40.197978 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:40.198525 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:40.199151 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:40.200012 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:40.200281 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:40.200367 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:40.204096 (+ 3729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.129:35535: BlockingRecv error: recv got EOF from 127.9.77.129:35535 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":104,"threads_started":1}
Aug 10 01:47:40 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:40 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:40.441186 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:40.434482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.130:51375)
0810 01:47:40.434866 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:40.434880 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:40.435057 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:40.435383 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:40.435386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:40.435667 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:40.435790 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:40.435797 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:40.436741 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:40.436746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:40.437378 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:40.437387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:40.437904 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:40.437908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:40.438018 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:40.438716 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:40.438729 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:40.440963 (+ 2234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":136,"threads_started":1}
W20260810 01:47:40.441248 23930 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:40.434564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:51375 (local address 127.9.77.190:45897)
0810 01:47:40.434755 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:40.434760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:40.434984 (+ 224us) server_negotiation.cc:407] Connection header received
0810 01:47:40.435182 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:40.435184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:40.435254 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:40.435351 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:40.435920 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:40.436621 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:40.437517 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:40.437774 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:40.437895 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:40.441118 (+ 3223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.130:51375: BlockingRecv error: recv got EOF from 127.9.77.130:51375 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 01:47:40 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:40 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:40.651787 23934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:40.644258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.131:44765)
0810 01:47:40.644797 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:40.644815 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:40.645019 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:40.645398 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:40.645402 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:40.645707 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:40.645852 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:40.645859 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:40.646954 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:40.646959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:40.647644 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:40.647654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:40.648298 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:40.648302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:40.648409 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:40.649127 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:40.649143 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:40.651620 (+ 2477us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":207,"threads_started":1}
W20260810 01:47:40.652079 23930 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:40.644494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:44765 (local address 127.9.77.190:45897)
0810 01:47:40.644723 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:40.644729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:40.644926 (+ 197us) server_negotiation.cc:407] Connection header received
0810 01:47:40.645177 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:40.645180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:40.645248 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:40.645366 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:40.646005 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:40.646815 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:40.647791 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:40.648156 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:40.648301 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:40.651866 (+ 3565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.131:44765: BlockingRecv error: recv got EOF from 127.9.77.131:44765 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 10 01:47:41 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:41 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:41.212281 23935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:41.205187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.129:53893)
0810 01:47:41.205583 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:41.205599 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:41.205717 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:41.206071 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:41.206073 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:41.206308 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:41.206437 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:41.206443 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:41.207364 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:41.207368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:41.208156 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:41.208171 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:41.208754 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:41.208758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:41.208859 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:41.209557 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:41.209571 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:41.212027 (+ 2456us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":118,"threads_started":1}
W20260810 01:47:41.212430 23936 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:41.205254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:53893 (local address 127.9.77.190:45897)
0810 01:47:41.205635 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:41.205639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:41.205655 (+ 16us) server_negotiation.cc:407] Connection header received
0810 01:47:41.205852 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:41.205855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:41.205919 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:41.206029 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:41.206566 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:41.207218 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:41.208293 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:41.208614 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:41.208742 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:41.212241 (+ 3499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.129:53893: BlockingRecv error: recv got EOF from 127.9.77.129:53893 (error 108)
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":142,"threads_started":1}
Aug 10 01:47:41 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:41 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:41.449208 23937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:41.442396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.130:43061)
0810 01:47:41.442756 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:41.442777 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:41.442917 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:41.443267 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:41.443271 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:41.443537 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:41.443673 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:41.443679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:41.444589 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:41.444593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:41.445238 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:41.445249 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:41.445899 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:41.445903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:41.446007 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:41.446710 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:41.446723 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:41.449044 (+ 2321us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":137,"threads_started":1}
W20260810 01:47:41.449343 23936 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:41.442627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:43061 (local address 127.9.77.190:45897)
0810 01:47:41.442794 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:41.442799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:41.442866 (+ 67us) server_negotiation.cc:407] Connection header received
0810 01:47:41.443009 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:41.443011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:41.443070 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:41.443153 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:41.443813 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:41.444463 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:41.445431 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:41.445747 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:41.445870 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:41.449205 (+ 3335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.130:43061: BlockingRecv error: recv got EOF from 127.9.77.130:43061 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 01:47:41 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:41 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:41.659121 23940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:41.652843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.131:49227)
0810 01:47:41.653200 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:41.653216 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:41.653358 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:41.653743 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:41.653746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:41.653980 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:41.654098 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:41.654103 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:41.654945 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:41.654948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:41.655580 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:41.655589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:41.656099 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:41.656103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:41.656176 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:41.656808 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:41.656821 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:41.658946 (+ 2125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":134,"threads_started":1}
W20260810 01:47:41.659198 23936 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:41.652896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:49227 (local address 127.9.77.190:45897)
0810 01:47:41.653091 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:41.653096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:41.653320 (+ 224us) server_negotiation.cc:407] Connection header received
0810 01:47:41.653541 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:41.653544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:41.653607 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:41.653688 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:41.654203 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:41.654817 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:41.655686 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:41.655938 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:41.656045 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:41.659086 (+ 3041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.131:49227: BlockingRecv error: recv got EOF from 127.9.77.131:49227 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 10 01:47:42 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:42 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:42.220168 23941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:42.213382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.129:46583)
0810 01:47:42.213784 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:42.213800 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:42.213926 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:42.214265 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:42.214268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:42.214518 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:42.214645 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:42.214651 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:42.215563 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:42.215566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:42.216275 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:42.216286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:42.216837 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:42.216840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:42.216924 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:42.217665 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:42.217678 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:42.219929 (+ 2251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":129,"threads_started":1}
W20260810 01:47:42.220388 23943 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:42.213459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:46583 (local address 127.9.77.190:45897)
0810 01:47:42.213849 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:42.213852 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:42.213868 (+ 16us) server_negotiation.cc:407] Connection header received
0810 01:47:42.214032 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:42.214035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:42.214088 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:42.214179 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:42.214756 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:42.215421 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:42.216420 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:42.216684 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:42.216784 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:42.220208 (+ 3424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.129:46583: BlockingRecv error: recv got EOF from 127.9.77.129:46583 (error 108)
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":83,"threads_started":1}
Aug 10 01:47:42 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:42 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:42.457036 23949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:42.450309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.130:55639)
0810 01:47:42.450756 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:42.450778 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:42.450968 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:42.451303 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:42.451306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:42.451613 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:42.451736 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:42.451742 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:42.452691 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:42.452696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:42.453341 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:42.453349 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:42.453858 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:42.453862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:42.453965 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:42.454572 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:42.454584 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:42.456791 (+ 2207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":189,"threads_started":1}
W20260810 01:47:42.457072 23943 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:42.450349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:55639 (local address 127.9.77.190:45897)
0810 01:47:42.450573 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:42.450579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:42.450846 (+ 267us) server_negotiation.cc:407] Connection header received
0810 01:47:42.451081 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:42.451084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:42.451158 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:42.451280 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:42.451878 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:42.452571 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:42.453476 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:42.453730 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:42.453842 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:42.456943 (+ 3101us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.130:55639: BlockingRecv error: recv got EOF from 127.9.77.130:55639 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 10 01:47:42 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:42 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.131@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:42.667085 23950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:42.660298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.131:43259)
0810 01:47:42.660679 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:42.660694 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:42.660883 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:42.661206 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:42.661209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:42.661489 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:42.661614 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:42.661620 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:42.662485 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:42.662491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:42.663148 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:42.663156 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:42.663692 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:42.663695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:42.663824 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:42.664615 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:42.664632 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:42.666913 (+ 2281us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":128,"threads_started":1}
W20260810 01:47:42.667213 23943 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:42.660348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:43259 (local address 127.9.77.190:45897)
0810 01:47:42.660554 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:42.660559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:42.660784 (+ 225us) server_negotiation.cc:407] Connection header received
0810 01:47:42.661004 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:42.661007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:42.661081 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:42.661177 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:42.661748 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:42.662367 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:42.663279 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:42.663562 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:42.663677 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:42.667073 (+ 3396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.131:43259: BlockingRecv error: recv got EOF from 127.9.77.131:43259 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 01:47:43 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:43 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.129@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:43.228945 23951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:43.221856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.129:42683)
0810 01:47:43.222383 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:43.222398 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:43.222515 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:43.222873 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:43.222876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:43.223191 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:43.223356 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:43.223364 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:43.224322 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:43.224327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:43.224976 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:43.224984 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:43.225498 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:43.225502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:43.225603 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:43.226379 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:43.226394 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:43.228708 (+ 2314us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":140,"threads_started":1}
W20260810 01:47:43.229079 23952 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:43.221916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:42683 (local address 127.9.77.190:45897)
0810 01:47:43.222388 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:43.222396 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:43.222482 (+ 86us) server_negotiation.cc:407] Connection header received
0810 01:47:43.222625 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:43.222628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:43.222689 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:43.222843 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:43.223489 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:43.224197 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:43.225108 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:43.225364 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:43.225479 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:43.228882 (+ 3403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.129:42683: BlockingRecv error: recv got EOF from 127.9.77.129:42683 (error 108)
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":119,"threads_started":1}
Aug 10 01:47:43 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 01:47:43 dist-test-slave-kvfs krb5kdc[22123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.77.130@KRBTEST.COM for oryx/127.9.77.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 01:47:43.469565 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:43.458400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45897 (local address 127.9.77.130:45315)
0810 01:47:43.458764 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:43.458781 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:43.458908 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:43.459235 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:43.459238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:43.459534 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:43.459698 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:43.459706 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:43.460880 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:43.460884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:43.461524 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:43.461532 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:43.461973 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:43.461975 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:43.462044 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:43.466958 (+ 4914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:43.466973 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:43.469336 (+ 2363us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.77.190:45897: Server oryx/127.9.77.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":101,"threads_started":1}
W20260810 01:47:43.469705 23952 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:43.458517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:45315 (local address 127.9.77.190:45897)
0810 01:47:43.458755 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:43.458760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:43.458867 (+ 107us) server_negotiation.cc:407] Connection header received
0810 01:47:43.459021 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:43.459024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:43.459080 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:43.459156 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:43.459847 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:43.460749 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:43.461629 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:43.461860 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:43.461947 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:43.469548 (+ 7601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.77.130:45315: BlockingRecv error: recv got EOF from 127.9.77.130:45315 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
I20260810 01:47:43.531204 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 23132
I20260810 01:47:43.558419 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 23371
I20260810 01:47:43.573169 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 23620
I20260810 01:47:43.593547 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 23015
2026-08-10T01:47:43Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18854 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T01:47:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:47:43Z Disabled control of system clock
I20260810 01:47:43.640426 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:43037
--webserver_interface=127.9.77.190
--webserver_port=0
--builtin_ntp_servers=127.9.77.148:34943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:43037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 01:47:43.776378 23963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:43.776751 23963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:43.776789 23963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:43.780937 23963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:47:43.781023 23963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:43.781037 23963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:43.781054 23963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:47:43.781071 23963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:47:43.786181 23963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:34943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:43037
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:43037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:43.787719 23963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:43.789537 23963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:43.800767 23970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:43.800868 23974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:43.800808 23971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:43.800796 23963 server_base.cc:1034] running on GCE node
I20260810 01:47:43.801827 23963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:43.803377 23963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:43.804585 23963 hybrid_clock.cc:648] HybridClock initialized: now 1786326463804553 us; error 47 us; skew 500 ppm
I20260810 01:47:43.807017 23963 webserver.cc:460] Webserver started at http://127.9.77.190:43899/ using document root <none> and password file <none>
I20260810 01:47:43.807822 23963 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:43.807931 23963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:43.808264 23963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:43.810976 23963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "ffa66083e06842c3973a56f2df78463d"
format_stamp: "Formatted at 2026-08-10 01:47:43 on dist-test-slave-kvfs"
server_key: "d238bb6d3bd12103d0002fb07a5034b3"
server_key_iv: "dc8edcafbe25ab98c44eb4c87c0bb181"
server_key_version: "encryptionkey@0"
I20260810 01:47:43.811676 23963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "ffa66083e06842c3973a56f2df78463d"
format_stamp: "Formatted at 2026-08-10 01:47:43 on dist-test-slave-kvfs"
server_key: "d238bb6d3bd12103d0002fb07a5034b3"
server_key_iv: "dc8edcafbe25ab98c44eb4c87c0bb181"
server_key_version: "encryptionkey@0"
I20260810 01:47:43.816360 23963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 01:47:43.819435 23982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:43.821053 23963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:47:43.821223 23963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "ffa66083e06842c3973a56f2df78463d"
format_stamp: "Formatted at 2026-08-10 01:47:43 on dist-test-slave-kvfs"
server_key: "d238bb6d3bd12103d0002fb07a5034b3"
server_key_iv: "dc8edcafbe25ab98c44eb4c87c0bb181"
server_key_version: "encryptionkey@0"
I20260810 01:47:43.821374 23963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:43.837549 23963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:43.838562 23963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:43.838821 23963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:43.847750 23963 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:43037
I20260810 01:47:43.847772 24072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:43037 every 8 connection(s)
I20260810 01:47:43.849133 23963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:47:43.853585 24073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:43.858932 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 23963
I20260810 01:47:43.859119 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:47:43.859498 9526 external_mini_cluster.cc:1428] Setting key f812914711fb0b29fa2a059a507a1e99
I20260810 01:47:43.861661 24073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffa66083e06842c3973a56f2df78463d: Bootstrap starting.
I20260810 01:47:43.864763 24073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffa66083e06842c3973a56f2df78463d: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:43.865724 24073 log.cc:824] T 00000000000000000000000000000000 P ffa66083e06842c3973a56f2df78463d: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:43.868619 24073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffa66083e06842c3973a56f2df78463d: No bootstrap required, opened a new log
I20260810 01:47:43.871088 24076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:43.861474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56142 (local address 127.9.77.190:43037)
0810 01:47:43.862295 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:43.862318 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:43.862365 (+ 47us) server_negotiation.cc:407] Connection header received
0810 01:47:43.863179 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:43.863200 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:43.863720 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:43.864148 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:43.864616 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:43.865796 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:43.867011 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:43.867573 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:43.867710 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:43.868877 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:43.868883 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:43.868899 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:47:43.869396 (+ 497us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:43.869483 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:47:43.869507 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:43.869599 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:43.869781 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 01:47:43.870154 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":169,"threads_started":1}
I20260810 01:47:43.873207 24073 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffa66083e06842c3973a56f2df78463d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa66083e06842c3973a56f2df78463d" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 43037 } }
I20260810 01:47:43.873553 24073 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffa66083e06842c3973a56f2df78463d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:43.873620 24073 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffa66083e06842c3973a56f2df78463d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffa66083e06842c3973a56f2df78463d, State: Initialized, Role: FOLLOWER
I20260810 01:47:43.874174 24073 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffa66083e06842c3973a56f2df78463d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa66083e06842c3973a56f2df78463d" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 43037 } }
I20260810 01:47:43.874327 24073 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffa66083e06842c3973a56f2df78463d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:47:43.874385 24073 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffa66083e06842c3973a56f2df78463d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:47:43.874469 24073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffa66083e06842c3973a56f2df78463d [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:43.875636 24073 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffa66083e06842c3973a56f2df78463d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa66083e06842c3973a56f2df78463d" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 43037 } }
I20260810 01:47:43.876008 24073 leader_election.cc:302] T 00000000000000000000000000000000 P ffa66083e06842c3973a56f2df78463d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffa66083e06842c3973a56f2df78463d; no voters:
I20260810 01:47:43.876544 24073 leader_election.cc:288] T 00000000000000000000000000000000 P ffa66083e06842c3973a56f2df78463d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:47:43.876654 24080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffa66083e06842c3973a56f2df78463d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:47:43.876849 24080 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffa66083e06842c3973a56f2df78463d [term 1 LEADER]: Becoming Leader. State: Replica: ffa66083e06842c3973a56f2df78463d, State: Running, Role: LEADER
I20260810 01:47:43.877282 24080 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffa66083e06842c3973a56f2df78463d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa66083e06842c3973a56f2df78463d" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 43037 } }
I20260810 01:47:43.878190 24073 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffa66083e06842c3973a56f2df78463d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:47:43.879441 24081 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffa66083e06842c3973a56f2df78463d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffa66083e06842c3973a56f2df78463d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa66083e06842c3973a56f2df78463d" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 43037 } } }
I20260810 01:47:43.879619 24081 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffa66083e06842c3973a56f2df78463d [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:43.880204 24082 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffa66083e06842c3973a56f2df78463d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffa66083e06842c3973a56f2df78463d. Latest consensus state: current_term: 1 leader_uuid: "ffa66083e06842c3973a56f2df78463d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa66083e06842c3973a56f2df78463d" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 43037 } } }
I20260810 01:47:43.880331 24082 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffa66083e06842c3973a56f2df78463d [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:43.881431 24089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:47:43.885245 24089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:47:43.892079 24089 catalog_manager.cc:1347] Generated new cluster ID: e087ef6060e84579bcd72c5082b86b32
I20260810 01:47:43.892184 24089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:47:43.901558 24089 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:47:43.902885 24089 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:47:43.910355 24089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffa66083e06842c3973a56f2df78463d: Generated new TSK 0
I20260810 01:47:43.911303 24089 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:47:43.919591 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:0
--local_ip_for_outbound_sockets=127.9.77.129
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:43037
--builtin_ntp_servers=127.9.77.148:34943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 01:47:44.061209 24110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:44.061596 24110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:44.061638 24110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:44.066028 24110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:44.066107 24110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:44.066167 24110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:47:44.071426 24110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:34943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:43037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:44.073051 24110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:44.074924 24110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:44.087460 24118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:44.087683 24117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:44.087441 24121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:44.088111 24110 server_base.cc:1034] running on GCE node
I20260810 01:47:44.088581 24110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:44.089442 24110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:44.090670 24110 hybrid_clock.cc:648] HybridClock initialized: now 1786326464090646 us; error 37 us; skew 500 ppm
I20260810 01:47:44.093205 24110 webserver.cc:460] Webserver started at http://127.9.77.129:33295/ using document root <none> and password file <none>
I20260810 01:47:44.093900 24110 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:44.093992 24110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:44.094205 24110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:44.096858 24110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/data/instance:
uuid: "80b454cfd8da437fa8ea853b600ff1cc"
format_stamp: "Formatted at 2026-08-10 01:47:44 on dist-test-slave-kvfs"
server_key: "9fc2addd25a291764fe9d5640120b1b1"
server_key_iv: "e4995e220cdfcb71c7b39533910e246c"
server_key_version: "encryptionkey@0"
I20260810 01:47:44.097611 24110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance:
uuid: "80b454cfd8da437fa8ea853b600ff1cc"
format_stamp: "Formatted at 2026-08-10 01:47:44 on dist-test-slave-kvfs"
server_key: "9fc2addd25a291764fe9d5640120b1b1"
server_key_iv: "e4995e220cdfcb71c7b39533910e246c"
server_key_version: "encryptionkey@0"
I20260810 01:47:44.102638 24110 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 01:47:44.106310 24131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:44.107997 24110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 01:47:44.108248 24110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "80b454cfd8da437fa8ea853b600ff1cc"
format_stamp: "Formatted at 2026-08-10 01:47:44 on dist-test-slave-kvfs"
server_key: "9fc2addd25a291764fe9d5640120b1b1"
server_key_iv: "e4995e220cdfcb71c7b39533910e246c"
server_key_version: "encryptionkey@0"
I20260810 01:47:44.108409 24110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:44.121199 24110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:44.122128 24110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:44.122539 24110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:44.123327 24110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:44.124644 24110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:44.124718 24110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:44.124809 24110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:44.124858 24110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:44.136247 24110 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:39913
I20260810 01:47:44.136307 24304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:39913 every 8 connection(s)
I20260810 01:47:44.137418 24110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:47:44.140146 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 24110
I20260810 01:47:44.140302 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance
I20260810 01:47:44.140673 9526 external_mini_cluster.cc:1428] Setting key b5e887f70f88bb5c65c3ff4e2b0a9b9b
I20260810 01:47:44.143357 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:0
--local_ip_for_outbound_sockets=127.9.77.130
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:43037
--builtin_ntp_servers=127.9.77.148:34943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 01:47:44.150624 24076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:44.139890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:50781 (local address 127.9.77.190:43037)
0810 01:47:44.140128 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:44.140135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:44.141735 (+ 1600us) server_negotiation.cc:407] Connection header received
0810 01:47:44.143951 (+ 2216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:44.143954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:44.144075 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:44.144171 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:44.145722 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:44.146687 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:44.148131 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:44.148416 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:44.148550 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:44.149996 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:44.150000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:44.150003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:47:44.150026 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:44.150053 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:47:44.150062 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:44.150146 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:44.150344 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 01:47:44.150431 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 01:47:44.151793 24309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:44.140564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:43037 (local address 127.9.77.129:50781)
0810 01:47:44.141541 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:44.141605 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:44.143628 (+ 2023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:44.144446 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:44.144460 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:44.144604 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:44.145509 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:44.145537 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:44.146875 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:44.146878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:44.147932 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:44.147942 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:44.148615 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:44.148619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:44.148842 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:44.149652 (+ 810us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 01:47:44.149654 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:44.149709 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 01:47:44.149795 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:44.150148 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:44.150162 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:44.150169 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 01:47:44.150331 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 01:47:44.150718 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":188,"threads_started":1}
I20260810 01:47:44.153614 24305 heartbeater.cc:347] Connected to a master server at 127.9.77.190:43037
I20260810 01:47:44.153926 24305 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:44.154637 24305 heartbeater.cc:511] Master 127.9.77.190:43037 requested a full tablet report, sending...
I20260810 01:47:44.156955 24011 ts_manager.cc:194] Registered new tserver with Master: 80b454cfd8da437fa8ea853b600ff1cc (127.9.77.129:39913)
I20260810 01:47:44.159102 24011 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.77.129:50781
W20260810 01:47:44.284341 24312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:44.284673 24312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:44.284713 24312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:44.288981 24312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:44.289054 24312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:44.289113 24312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:47:44.294232 24312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:34943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:43037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:44.295737 24312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:44.297618 24312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:44.310433 24320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:44.310566 24312 server_base.cc:1034] running on GCE node
W20260810 01:47:44.310438 24321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:44.310403 24329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:44.311350 24312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:44.312450 24312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:44.313633 24312 hybrid_clock.cc:648] HybridClock initialized: now 1786326464313592 us; error 57 us; skew 500 ppm
I20260810 01:47:44.316406 24312 webserver.cc:460] Webserver started at http://127.9.77.130:36015/ using document root <none> and password file <none>
I20260810 01:47:44.317202 24312 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:44.317286 24312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:44.317520 24312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:44.320618 24312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/data/instance:
uuid: "355b9926693c4bdda92e659c92a830e3"
format_stamp: "Formatted at 2026-08-10 01:47:44 on dist-test-slave-kvfs"
server_key: "89da63c8729fa86861aae5efc5fda50f"
server_key_iv: "50380482c3177ec9f932bae2a2c89aba"
server_key_version: "encryptionkey@0"
I20260810 01:47:44.321312 24312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance:
uuid: "355b9926693c4bdda92e659c92a830e3"
format_stamp: "Formatted at 2026-08-10 01:47:44 on dist-test-slave-kvfs"
server_key: "89da63c8729fa86861aae5efc5fda50f"
server_key_iv: "50380482c3177ec9f932bae2a2c89aba"
server_key_version: "encryptionkey@0"
I20260810 01:47:44.326545 24312 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 01:47:44.331032 24341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:44.332859 24312 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 01:47:44.333058 24312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "355b9926693c4bdda92e659c92a830e3"
format_stamp: "Formatted at 2026-08-10 01:47:44 on dist-test-slave-kvfs"
server_key: "89da63c8729fa86861aae5efc5fda50f"
server_key_iv: "50380482c3177ec9f932bae2a2c89aba"
server_key_version: "encryptionkey@0"
I20260810 01:47:44.333232 24312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:44.353590 24312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:44.354691 24312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:44.354893 24312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:44.355631 24312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:44.356906 24312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:44.356967 24312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:44.357018 24312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:44.357041 24312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:44.367743 24312 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:37947
I20260810 01:47:44.367820 24526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:37947 every 8 connection(s)
I20260810 01:47:44.369062 24312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:47:44.374842 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 24312
I20260810 01:47:44.375000 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance
I20260810 01:47:44.375322 9526 external_mini_cluster.cc:1428] Setting key a3f049e258b582424b80cfc5efd78f25
I20260810 01:47:44.377731 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:0
--local_ip_for_outbound_sockets=127.9.77.131
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:43037
--builtin_ntp_servers=127.9.77.148:34943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 01:47:44.381165 24076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:44.371650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:45333 (local address 127.9.77.190:43037)
0810 01:47:44.371820 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:44.371824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:44.373286 (+ 1462us) server_negotiation.cc:407] Connection header received
0810 01:47:44.374969 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:44.374973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:44.375047 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:44.375169 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:44.376435 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:44.377037 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:44.378464 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:44.378709 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:44.378837 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:44.380495 (+ 1658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:44.380499 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:44.380501 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:47:44.380516 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:44.380542 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:47:44.380551 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:44.380646 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:44.380896 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 01:47:44.380979 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 01:47:44.382475 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:44.372208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:43037 (local address 127.9.77.130:45333)
0810 01:47:44.373069 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:44.373121 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:44.374708 (+ 1587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:44.375402 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:44.375432 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:44.375567 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:44.376279 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:44.376299 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:44.377174 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:44.377178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:44.378304 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:44.378314 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:44.378887 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:44.378891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:44.379110 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:44.380088 (+ 978us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 01:47:44.380092 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:44.380167 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 01:47:44.380285 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:44.380663 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:44.380676 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:44.380684 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 01:47:44.380880 (+ 196us) client_negotiation.cc:240] Negotiation successful
0810 01:47:44.381251 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":158,"threads_started":1}
I20260810 01:47:44.383881 24527 heartbeater.cc:347] Connected to a master server at 127.9.77.190:43037
I20260810 01:47:44.384217 24527 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:44.384940 24527 heartbeater.cc:511] Master 127.9.77.190:43037 requested a full tablet report, sending...
I20260810 01:47:44.386328 24011 ts_manager.cc:194] Registered new tserver with Master: 355b9926693c4bdda92e659c92a830e3 (127.9.77.130:37947)
I20260810 01:47:44.387066 24011 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.77.130:45333
W20260810 01:47:44.521654 24531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:44.522291 24531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:44.522403 24531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:44.527312 24531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:44.527377 24531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:44.527503 24531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:47:44.532804 24531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:34943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:43037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:44.534224 24531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:44.536000 24531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 01:47:44.548158 24531 server_base.cc:1034] running on GCE node
W20260810 01:47:44.548103 24543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:44.548137 24545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:44.548103 24542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:44.549003 24531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:44.549841 24531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:44.551061 24531 hybrid_clock.cc:648] HybridClock initialized: now 1786326464551040 us; error 38 us; skew 500 ppm
I20260810 01:47:44.553643 24531 webserver.cc:460] Webserver started at http://127.9.77.131:40319/ using document root <none> and password file <none>
I20260810 01:47:44.554417 24531 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:44.554497 24531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:44.554817 24531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:44.557490 24531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/data/instance:
uuid: "1e5f40717cda47ac9b3f59cf1499edc2"
format_stamp: "Formatted at 2026-08-10 01:47:44 on dist-test-slave-kvfs"
server_key: "cce68435a730999605d26171a9fc96c8"
server_key_iv: "46d13c59d61c463199b839496d03f8fc"
server_key_version: "encryptionkey@0"
I20260810 01:47:44.558190 24531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance:
uuid: "1e5f40717cda47ac9b3f59cf1499edc2"
format_stamp: "Formatted at 2026-08-10 01:47:44 on dist-test-slave-kvfs"
server_key: "cce68435a730999605d26171a9fc96c8"
server_key_iv: "46d13c59d61c463199b839496d03f8fc"
server_key_version: "encryptionkey@0"
I20260810 01:47:44.562870 24531 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 01:47:44.566113 24553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:44.567740 24531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 01:47:44.567909 24531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "1e5f40717cda47ac9b3f59cf1499edc2"
format_stamp: "Formatted at 2026-08-10 01:47:44 on dist-test-slave-kvfs"
server_key: "cce68435a730999605d26171a9fc96c8"
server_key_iv: "46d13c59d61c463199b839496d03f8fc"
server_key_version: "encryptionkey@0"
I20260810 01:47:44.568143 24531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:44.589761 24531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:44.590723 24531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:44.590976 24531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:44.591759 24531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:44.593144 24531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:44.593235 24531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:44.593343 24531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:44.593394 24531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:44.604398 24531 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:40075
I20260810 01:47:44.604429 24741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:40075 every 8 connection(s)
I20260810 01:47:44.605578 24531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:47:44.608771 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 24531
I20260810 01:47:44.608922 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance
I20260810 01:47:44.609311 9526 external_mini_cluster.cc:1428] Setting key e6ccae1f8d1ab3bc2ff84b5b83d6bce2
I20260810 01:47:44.617170 24076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:44.607928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:36657 (local address 127.9.77.190:43037)
0810 01:47:44.608148 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:44.608152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:44.609655 (+ 1503us) server_negotiation.cc:407] Connection header received
0810 01:47:44.611728 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:44.611732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:44.611805 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:44.611885 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:44.613080 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:44.613744 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:44.614979 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:44.615217 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:44.615301 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:44.616679 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:44.616682 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:44.616685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:47:44.616699 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:44.616723 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:47:44.616731 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:44.616808 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:44.616964 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 01:47:44.617029 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 01:47:44.618289 24747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:44.608473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:43037 (local address 127.9.77.131:36657)
0810 01:47:44.609478 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:44.609529 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:44.611410 (+ 1881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:44.612168 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:44.612176 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:44.612281 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:44.612900 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:44.612914 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:44.613864 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:44.613867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:44.614864 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:44.614874 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:44.615364 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:44.615367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:44.615546 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:44.616378 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 01:47:44.616380 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:44.616432 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 01:47:44.616517 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:44.616817 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:44.616824 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:44.616829 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 01:47:44.616932 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 01:47:44.617219 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":194,"threads_started":1}
I20260810 01:47:44.619647 24744 heartbeater.cc:347] Connected to a master server at 127.9.77.190:43037
I20260810 01:47:44.619939 24744 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:44.620652 24744 heartbeater.cc:511] Master 127.9.77.190:43037 requested a full tablet report, sending...
I20260810 01:47:44.622042 24011 ts_manager.cc:194] Registered new tserver with Master: 1e5f40717cda47ac9b3f59cf1499edc2 (127.9.77.131:40075)
I20260810 01:47:44.622849 24011 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.77.131:36657
I20260810 01:47:44.624130 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 01:47:44.627655 24076 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:44.626590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56148 (local address 127.9.77.190:43037)
0810 01:47:44.626749 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:44.626752 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:44.626977 (+ 225us) server_negotiation.cc:407] Connection header received
0810 01:47:44.627089 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:44.627091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:44.627152 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:44.627221 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:44.627492 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56148: BlockingRecv error: recv got EOF from 127.0.0.1:56148 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 01:47:44.628008 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 24110
I20260810 01:47:44.646628 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 24312
I20260810 01:47:44.662377 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 24531
I20260810 01:47:44.674904 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 23963
2026-08-10T01:47:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1076 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T01:47:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:47:44Z Disabled control of system clock
I20260810 01:47:44.716959 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:41897
--webserver_interface=127.9.77.190
--webserver_port=0
--builtin_ntp_servers=127.9.77.148:36163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:41897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 01:47:44.853216 24757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:44.853709 24757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:44.853756 24757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:44.858158 24757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:47:44.858254 24757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:44.858271 24757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:44.858289 24757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:47:44.858309 24757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:47:44.863867 24757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:36163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:41897
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:41897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:44.865523 24757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:44.867426 24757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:44.878016 24769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:44.878016 24771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:44.878079 24757 server_base.cc:1034] running on GCE node
W20260810 01:47:44.878048 24768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:44.879173 24757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:44.880697 24757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:44.881924 24757 hybrid_clock.cc:648] HybridClock initialized: now 1786326464881907 us; error 34 us; skew 500 ppm
I20260810 01:47:44.884311 24757 webserver.cc:460] Webserver started at http://127.9.77.190:43451/ using document root <none> and password file <none>
I20260810 01:47:44.885049 24757 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:44.885151 24757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:44.885435 24757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:44.888001 24757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "aa641cfb864848e997ff66311ba8fa8b"
format_stamp: "Formatted at 2026-08-10 01:47:44 on dist-test-slave-kvfs"
server_key: "251afaf3a23632a049e59043372cdec7"
server_key_iv: "675640ae713b0beed2b65c7cb03b87f0"
server_key_version: "encryptionkey@0"
I20260810 01:47:44.888761 24757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "aa641cfb864848e997ff66311ba8fa8b"
format_stamp: "Formatted at 2026-08-10 01:47:44 on dist-test-slave-kvfs"
server_key: "251afaf3a23632a049e59043372cdec7"
server_key_iv: "675640ae713b0beed2b65c7cb03b87f0"
server_key_version: "encryptionkey@0"
I20260810 01:47:44.893405 24757 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 01:47:44.896579 24783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:44.898185 24757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:47:44.898357 24757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "aa641cfb864848e997ff66311ba8fa8b"
format_stamp: "Formatted at 2026-08-10 01:47:44 on dist-test-slave-kvfs"
server_key: "251afaf3a23632a049e59043372cdec7"
server_key_iv: "675640ae713b0beed2b65c7cb03b87f0"
server_key_version: "encryptionkey@0"
I20260810 01:47:44.898515 24757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:44.904237 24757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:44.905045 24757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:44.905259 24757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:44.913415 24757 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:41897
I20260810 01:47:44.913445 24867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:41897 every 8 connection(s)
I20260810 01:47:44.914667 24757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:47:44.919018 24868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:44.924957 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 24757
I20260810 01:47:44.925163 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:47:44.925570 9526 external_mini_cluster.cc:1428] Setting key 0f30d0d9881c188a63cfba691d06f4ed
I20260810 01:47:44.927135 24868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa641cfb864848e997ff66311ba8fa8b: Bootstrap starting.
I20260810 01:47:44.930296 24868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa641cfb864848e997ff66311ba8fa8b: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:44.931593 24868 log.cc:824] T 00000000000000000000000000000000 P aa641cfb864848e997ff66311ba8fa8b: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:44.934315 24871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:44.927590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49842 (local address 127.9.77.190:41897)
0810 01:47:44.928462 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:44.928483 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:44.928530 (+ 47us) server_negotiation.cc:407] Connection header received
0810 01:47:44.929359 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:44.929381 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:44.929853 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:44.930240 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:44.932040 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:44.932051 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:44.932080 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:47:44.932118 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:44.932214 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:47:44.932267 (+ 53us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:44.932499 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:44.932709 (+ 210us) server_negotiation.cc:299] Negotiation successful
0810 01:47:44.933081 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":178,"threads_started":1}
I20260810 01:47:44.936718 24868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa641cfb864848e997ff66311ba8fa8b: No bootstrap required, opened a new log
I20260810 01:47:44.941026 24868 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa641cfb864848e997ff66311ba8fa8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa641cfb864848e997ff66311ba8fa8b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 41897 } }
I20260810 01:47:44.941346 24868 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa641cfb864848e997ff66311ba8fa8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:44.941423 24868 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa641cfb864848e997ff66311ba8fa8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa641cfb864848e997ff66311ba8fa8b, State: Initialized, Role: FOLLOWER
I20260810 01:47:44.941915 24868 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa641cfb864848e997ff66311ba8fa8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa641cfb864848e997ff66311ba8fa8b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 41897 } }
I20260810 01:47:44.942052 24868 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa641cfb864848e997ff66311ba8fa8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:47:44.942096 24868 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa641cfb864848e997ff66311ba8fa8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:47:44.942181 24868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa641cfb864848e997ff66311ba8fa8b [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:44.943661 24868 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa641cfb864848e997ff66311ba8fa8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa641cfb864848e997ff66311ba8fa8b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 41897 } }
I20260810 01:47:44.944098 24868 leader_election.cc:302] T 00000000000000000000000000000000 P aa641cfb864848e997ff66311ba8fa8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa641cfb864848e997ff66311ba8fa8b; no voters:
I20260810 01:47:44.944446 24868 leader_election.cc:288] T 00000000000000000000000000000000 P aa641cfb864848e997ff66311ba8fa8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:47:44.944617 24873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa641cfb864848e997ff66311ba8fa8b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:47:44.944890 24873 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa641cfb864848e997ff66311ba8fa8b [term 1 LEADER]: Becoming Leader. State: Replica: aa641cfb864848e997ff66311ba8fa8b, State: Running, Role: LEADER
I20260810 01:47:44.945246 24873 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa641cfb864848e997ff66311ba8fa8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa641cfb864848e997ff66311ba8fa8b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 41897 } }
I20260810 01:47:44.946044 24868 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa641cfb864848e997ff66311ba8fa8b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:47:44.947333 24874 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa641cfb864848e997ff66311ba8fa8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa641cfb864848e997ff66311ba8fa8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa641cfb864848e997ff66311ba8fa8b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 41897 } } }
I20260810 01:47:44.947564 24874 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa641cfb864848e997ff66311ba8fa8b [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:44.948048 24887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:47:44.947333 24875 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa641cfb864848e997ff66311ba8fa8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa641cfb864848e997ff66311ba8fa8b. Latest consensus state: current_term: 1 leader_uuid: "aa641cfb864848e997ff66311ba8fa8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa641cfb864848e997ff66311ba8fa8b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 41897 } } }
I20260810 01:47:44.948414 24875 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa641cfb864848e997ff66311ba8fa8b [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:44.952253 24887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:47:44.959506 24887 catalog_manager.cc:1347] Generated new cluster ID: d629edd5791447bb94b2a10cefbe518e
I20260810 01:47:44.959604 24887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:47:44.977641 24887 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:47:44.978111 24887 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:47:44.983485 24887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa641cfb864848e997ff66311ba8fa8b: Generated new TSK 0
I20260810 01:47:44.984436 24887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:47:44.993068 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:0
--local_ip_for_outbound_sockets=127.9.77.129
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:41897
--builtin_ntp_servers=127.9.77.148:36163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 01:47:45.129819 24903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:45.130169 24903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:45.130252 24903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:45.134537 24903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:45.134632 24903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:45.134727 24903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:47:45.139792 24903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:36163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:41897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:45.141306 24903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:45.143129 24903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:45.155324 24917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:45.155329 24923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:45.155329 24916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:45.155771 24903 server_base.cc:1034] running on GCE node
I20260810 01:47:45.156334 24903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:45.157136 24903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:45.158346 24903 hybrid_clock.cc:648] HybridClock initialized: now 1786326465158314 us; error 53 us; skew 500 ppm
I20260810 01:47:45.160676 24903 webserver.cc:460] Webserver started at http://127.9.77.129:41167/ using document root <none> and password file <none>
I20260810 01:47:45.161407 24903 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:45.161510 24903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:45.161788 24903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:45.164386 24903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/data/instance:
uuid: "0287dd70059743209a7e2bcb20941192"
format_stamp: "Formatted at 2026-08-10 01:47:45 on dist-test-slave-kvfs"
server_key: "f922f9066f30c8105fb98dabf285e208"
server_key_iv: "efcdab47041abb32391118743c6c2733"
server_key_version: "encryptionkey@0"
I20260810 01:47:45.165048 24903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance:
uuid: "0287dd70059743209a7e2bcb20941192"
format_stamp: "Formatted at 2026-08-10 01:47:45 on dist-test-slave-kvfs"
server_key: "f922f9066f30c8105fb98dabf285e208"
server_key_iv: "efcdab47041abb32391118743c6c2733"
server_key_version: "encryptionkey@0"
I20260810 01:47:45.169448 24903 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 01:47:45.172703 24930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:45.174297 24903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:47:45.174412 24903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "0287dd70059743209a7e2bcb20941192"
format_stamp: "Formatted at 2026-08-10 01:47:45 on dist-test-slave-kvfs"
server_key: "f922f9066f30c8105fb98dabf285e208"
server_key_iv: "efcdab47041abb32391118743c6c2733"
server_key_version: "encryptionkey@0"
I20260810 01:47:45.174532 24903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:45.179947 24903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:45.180709 24903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:45.180871 24903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:45.181466 24903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:45.182528 24903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:45.182576 24903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:45.182642 24903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:45.182673 24903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:45.193200 24903 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:35939
I20260810 01:47:45.193228 25121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:35939 every 8 connection(s)
I20260810 01:47:45.194319 24903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:47:45.201490 24871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:45.196629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:40567 (local address 127.9.77.190:41897)
0810 01:47:45.196812 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:45.196817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:45.198148 (+ 1331us) server_negotiation.cc:407] Connection header received
0810 01:47:45.199631 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:45.199633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:45.199683 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:45.199766 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:45.201022 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:45.201025 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:45.201028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:47:45.201041 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:45.201066 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:47:45.201076 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:45.201164 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:45.201290 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 01:47:45.201338 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 01:47:45.202641 25125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:45.197181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:41897 (local address 127.9.77.129:40567)
0810 01:47:45.197973 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:45.198020 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:45.199400 (+ 1380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:45.199959 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:45.199967 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:45.200090 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:45.200728 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 01:47:45.200729 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:45.200779 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 01:47:45.200860 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:45.201157 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:45.201165 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:45.201169 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 01:47:45.201328 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 01:47:45.201600 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":138,"threads_started":1}
I20260810 01:47:45.203220 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 24903
I20260810 01:47:45.203370 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance
I20260810 01:47:45.203830 9526 external_mini_cluster.cc:1428] Setting key d308d32c451ae23a7593a781d8afc822
I20260810 01:47:45.204382 25123 heartbeater.cc:347] Connected to a master server at 127.9.77.190:41897
I20260810 01:47:45.204669 25123 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:45.205194 25123 heartbeater.cc:511] Master 127.9.77.190:41897 requested a full tablet report, sending...
I20260810 01:47:45.206403 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:0
--local_ip_for_outbound_sockets=127.9.77.130
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:41897
--builtin_ntp_servers=127.9.77.148:36163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 01:47:45.207183 24808 ts_manager.cc:194] Registered new tserver with Master: 0287dd70059743209a7e2bcb20941192 (127.9.77.129:35939)
W20260810 01:47:45.351265 25126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:45.351610 25126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:45.351658 25126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:45.355986 25126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:45.356127 25126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:45.356202 25126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:47:45.361347 25126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:36163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:41897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:45.362794 25126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:45.364699 25126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:45.376724 25134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:45.376718 25136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:45.376770 25132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:45.377017 25126 server_base.cc:1034] running on GCE node
I20260810 01:47:45.377625 25126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:45.378520 25126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:45.379755 25126 hybrid_clock.cc:648] HybridClock initialized: now 1786326465379715 us; error 53 us; skew 500 ppm
I20260810 01:47:45.382381 25126 webserver.cc:460] Webserver started at http://127.9.77.130:41037/ using document root <none> and password file <none>
I20260810 01:47:45.383162 25126 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:45.383276 25126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:45.383572 25126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:45.386336 25126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/data/instance:
uuid: "88bb0780ce6644a8932e64cb52980d8e"
format_stamp: "Formatted at 2026-08-10 01:47:45 on dist-test-slave-kvfs"
server_key: "3c8e5a812e516fa9374196a6a2b92f72"
server_key_iv: "52e53b4484462a72e9eb787e9bcd7b8a"
server_key_version: "encryptionkey@0"
I20260810 01:47:45.387020 25126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance:
uuid: "88bb0780ce6644a8932e64cb52980d8e"
format_stamp: "Formatted at 2026-08-10 01:47:45 on dist-test-slave-kvfs"
server_key: "3c8e5a812e516fa9374196a6a2b92f72"
server_key_iv: "52e53b4484462a72e9eb787e9bcd7b8a"
server_key_version: "encryptionkey@0"
I20260810 01:47:45.391757 25126 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260810 01:47:45.395351 25147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:45.397182 25126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:47:45.397368 25126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "88bb0780ce6644a8932e64cb52980d8e"
format_stamp: "Formatted at 2026-08-10 01:47:45 on dist-test-slave-kvfs"
server_key: "3c8e5a812e516fa9374196a6a2b92f72"
server_key_iv: "52e53b4484462a72e9eb787e9bcd7b8a"
server_key_version: "encryptionkey@0"
I20260810 01:47:45.397534 25126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:45.403753 25126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:45.404649 25126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:45.404892 25126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:45.405614 25126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:45.406950 25126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:45.407030 25126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:45.407115 25126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:45.407176 25126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:45.418715 25126 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:38127
I20260810 01:47:45.418742 25341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:38127 every 8 connection(s)
I20260810 01:47:45.419919 25126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:47:45.427050 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 25126
I20260810 01:47:45.427227 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance
I20260810 01:47:45.427663 9526 external_mini_cluster.cc:1428] Setting key 16a470ab047b45831d6bbc8c88930558
I20260810 01:47:45.428113 24871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:45.422381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:35899 (local address 127.9.77.190:41897)
0810 01:47:45.422562 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:45.422566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:45.423901 (+ 1335us) server_negotiation.cc:407] Connection header received
0810 01:47:45.425653 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:45.425657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:45.425739 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:45.425838 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:45.427436 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:45.427441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:45.427457 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:47:45.427479 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:45.427516 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:47:45.427532 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:45.427656 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:45.427837 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 01:47:45.427908 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 01:47:45.429553 25345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:45.422815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:41897 (local address 127.9.77.130:35899)
0810 01:47:45.423685 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:45.423737 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:45.425353 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:45.426078 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:45.426087 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:45.426203 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:45.426988 (+ 785us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 01:47:45.426990 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:45.427043 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 01:47:45.427163 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:45.427637 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:45.427646 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:45.427651 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 01:47:45.427896 (+ 245us) client_negotiation.cc:240] Negotiation successful
0810 01:47:45.428253 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":133,"threads_started":1}
I20260810 01:47:45.430474 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:0
--local_ip_for_outbound_sockets=127.9.77.131
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:41897
--builtin_ntp_servers=127.9.77.148:36163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 01:47:45.431097 25342 heartbeater.cc:347] Connected to a master server at 127.9.77.190:41897
I20260810 01:47:45.431421 25342 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:45.432081 25342 heartbeater.cc:511] Master 127.9.77.190:41897 requested a full tablet report, sending...
I20260810 01:47:45.433605 24808 ts_manager.cc:194] Registered new tserver with Master: 88bb0780ce6644a8932e64cb52980d8e (127.9.77.130:38127)
W20260810 01:47:45.572171 25346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:45.572579 25346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:45.572662 25346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:45.577014 25346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:45.577126 25346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:45.577225 25346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:47:45.582520 25346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:36163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:41897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:45.584118 25346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:45.585958 25346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:45.597368 25353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:45.597368 25356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:45.597394 25354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:45.597436 25346 server_base.cc:1034] running on GCE node
I20260810 01:47:45.598096 25346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:45.598811 25346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:45.600003 25346 hybrid_clock.cc:648] HybridClock initialized: now 1786326465599983 us; error 31 us; skew 500 ppm
I20260810 01:47:45.602545 25346 webserver.cc:460] Webserver started at http://127.9.77.131:36849/ using document root <none> and password file <none>
I20260810 01:47:45.603430 25346 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:45.603506 25346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:45.603788 25346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:45.606465 25346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/data/instance:
uuid: "917be639f76e42b3abcc59dc7df13eea"
format_stamp: "Formatted at 2026-08-10 01:47:45 on dist-test-slave-kvfs"
server_key: "17a68a3238cfc0e752ef8985ff89a8a8"
server_key_iv: "c61f1a4119b9a9cf33ede354a23a659c"
server_key_version: "encryptionkey@0"
I20260810 01:47:45.607138 25346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance:
uuid: "917be639f76e42b3abcc59dc7df13eea"
format_stamp: "Formatted at 2026-08-10 01:47:45 on dist-test-slave-kvfs"
server_key: "17a68a3238cfc0e752ef8985ff89a8a8"
server_key_iv: "c61f1a4119b9a9cf33ede354a23a659c"
server_key_version: "encryptionkey@0"
I20260810 01:47:45.611773 25346 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260810 01:47:45.614979 25369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:45.616582 25346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:47:45.616720 25346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "917be639f76e42b3abcc59dc7df13eea"
format_stamp: "Formatted at 2026-08-10 01:47:45 on dist-test-slave-kvfs"
server_key: "17a68a3238cfc0e752ef8985ff89a8a8"
server_key_iv: "c61f1a4119b9a9cf33ede354a23a659c"
server_key_version: "encryptionkey@0"
I20260810 01:47:45.616876 25346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:45.622907 25346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:45.623785 25346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:45.624006 25346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:45.624744 25346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:45.625890 25346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:45.625941 25346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:45.626011 25346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:45.626053 25346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:45.636309 25346 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:37033
I20260810 01:47:45.636334 25561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:37033 every 8 connection(s)
I20260810 01:47:45.637434 25346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:47:45.641706 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 25346
I20260810 01:47:45.641821 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance
I20260810 01:47:45.642175 9526 external_mini_cluster.cc:1428] Setting key 3d8ca01812e5eacd78c5a3afd5a38282
I20260810 01:47:45.645323 24871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:45.639896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:47195 (local address 127.9.77.190:41897)
0810 01:47:45.640135 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:45.640139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:45.641397 (+ 1258us) server_negotiation.cc:407] Connection header received
0810 01:47:45.643151 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:45.643154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:45.643216 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:45.643301 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:45.644858 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:45.644862 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:45.644866 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:47:45.644885 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:45.644909 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:47:45.644918 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:45.645001 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:45.645149 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 01:47:45.645194 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 01:47:45.646468 25565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:45.640378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:41897 (local address 127.9.77.131:47195)
0810 01:47:45.641208 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:45.641257 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:45.642923 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:45.643567 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:45.643577 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:45.643713 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:45.644509 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 01:47:45.644510 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:45.644560 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 01:47:45.644685 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:45.645009 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:45.645017 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:45.645022 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 01:47:45.645177 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 01:47:45.645452 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":129,"threads_started":1}
I20260810 01:47:45.648082 25563 heartbeater.cc:347] Connected to a master server at 127.9.77.190:41897
I20260810 01:47:45.648424 25563 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:45.648954 25563 heartbeater.cc:511] Master 127.9.77.190:41897 requested a full tablet report, sending...
I20260810 01:47:45.650135 24808 ts_manager.cc:194] Registered new tserver with Master: 917be639f76e42b3abcc59dc7df13eea (127.9.77.131:37033)
I20260810 01:47:45.656772 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 01:47:45.660552 24871 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:47:45.659554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49850 (local address 127.9.77.190:41897)
0810 01:47:45.659704 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:45.659707 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:45.659866 (+ 159us) server_negotiation.cc:407] Connection header received
0810 01:47:45.659989 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:45.659991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:45.660041 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:45.660113 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:45.660358 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49850: BlockingRecv error: recv got EOF from 127.0.0.1:49850 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 01:47:45.660840 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 24903
I20260810 01:47:45.690320 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 25126
I20260810 01:47:45.710615 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 25346
I20260810 01:47:45.739235 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 24757
2026-08-10T01:47:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (1065 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:47:45 dist-test-slave-kvfs krb5kdc[25579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:47:45 dist-test-slave-kvfs krb5kdc[25579](info): set up 2 sockets
Aug 10 01:47:45 dist-test-slave-kvfs krb5kdc[25579](info): commencing operation
krb5kdc: starting...
W20260810 01:47:47.035324 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.265s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:47:47 dist-test-slave-kvfs krb5kdc[25579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:47:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:47:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:47:47.183702 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:36293
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:39499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:36293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:47.320341 25601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:47.320741 25601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:47.320850 25601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:47.325019 25601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:47:47.325107 25601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:47.325145 25601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:47.325203 25601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:47:47.325235 25601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:47:47.331244 25601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:39499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:36293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:36293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:47.332794 25601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:47.334174 25601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:47.345142 25608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:47.345178 25607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:47.345153 25612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:47.345584 25601 server_base.cc:1034] running on GCE node
I20260810 01:47:47.346411 25601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:47.347860 25601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:47.349061 25601 hybrid_clock.cc:648] HybridClock initialized: now 1786326467349038 us; error 38 us; skew 500 ppm
Aug 10 01:47:47 dist-test-slave-kvfs krb5kdc[25579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326467, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:47.352551 25601 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:47:47.353691 25601 webserver.cc:460] Webserver started at http://127.9.77.190:40709/ using document root <none> and password file <none>
I20260810 01:47:47.354425 25601 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:47.354494 25601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:47.354763 25601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:47.357460 25601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "accbd4eff69a4d44b4559891278cad0d"
format_stamp: "Formatted at 2026-08-10 01:47:47 on dist-test-slave-kvfs"
server_key: "c3cec53206ca9e8b2e396851cfc2b63a"
server_key_iv: "966637bf6b48797c0d7a36a785afd5fc"
server_key_version: "encryptionkey@0"
I20260810 01:47:47.358174 25601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "accbd4eff69a4d44b4559891278cad0d"
format_stamp: "Formatted at 2026-08-10 01:47:47 on dist-test-slave-kvfs"
server_key: "c3cec53206ca9e8b2e396851cfc2b63a"
server_key_iv: "966637bf6b48797c0d7a36a785afd5fc"
server_key_version: "encryptionkey@0"
I20260810 01:47:47.363001 25601 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 01:47:47.365967 25620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:47.367411 25601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 01:47:47.367586 25601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "accbd4eff69a4d44b4559891278cad0d"
format_stamp: "Formatted at 2026-08-10 01:47:47 on dist-test-slave-kvfs"
server_key: "c3cec53206ca9e8b2e396851cfc2b63a"
server_key_iv: "966637bf6b48797c0d7a36a785afd5fc"
server_key_version: "encryptionkey@0"
I20260810 01:47:47.367719 25601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:47.404645 25601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:47.408994 25601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:47.409248 25601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:47.417945 25601 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:36293
I20260810 01:47:47.418304 25702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:36293 every 8 connection(s)
I20260810 01:47:47.419245 25601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:47:47.422941 25704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:47.423367 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 25601
I20260810 01:47:47.423522 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:47:47.423893 9526 external_mini_cluster.cc:1428] Setting key e9e4ef182ce0b4a10413427be5e89c10
I20260810 01:47:47.430904 25704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P accbd4eff69a4d44b4559891278cad0d: Bootstrap starting.
I20260810 01:47:47.434324 25704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P accbd4eff69a4d44b4559891278cad0d: Neither blocks nor log segments found. Creating new log.
Aug 10 01:47:47 dist-test-slave-kvfs krb5kdc[25579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:47:47.435267 25704 log.cc:824] T 00000000000000000000000000000000 P accbd4eff69a4d44b4559891278cad0d: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:47.438282 25704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P accbd4eff69a4d44b4559891278cad0d: No bootstrap required, opened a new log
I20260810 01:47:47.441928 25709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:47.425860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49460 (local address 127.9.77.190:36293)
0810 01:47:47.426381 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:47.426405 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:47.426644 (+ 239us) server_negotiation.cc:407] Connection header received
0810 01:47:47.427559 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:47.427579 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:47.427995 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:47.428459 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:47.429484 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:47.431088 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:47.432016 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:47.432841 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:47.433033 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:47.435911 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:47.435918 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:47.435940 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:47.435969 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:47.438114 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:47.438615 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:47.438621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:47.438628 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:47.438795 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:47.439146 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:47.439149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:47.439151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:47.439733 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:47.439946 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:47.440265 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 01:47:47.440632 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":162,"threads_started":1}
I20260810 01:47:47.443511 25704 raft_consensus.cc:348] T 00000000000000000000000000000000 P accbd4eff69a4d44b4559891278cad0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "accbd4eff69a4d44b4559891278cad0d" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 36293 } }
I20260810 01:47:47.443830 25704 raft_consensus.cc:374] T 00000000000000000000000000000000 P accbd4eff69a4d44b4559891278cad0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:47.443887 25704 raft_consensus.cc:729] T 00000000000000000000000000000000 P accbd4eff69a4d44b4559891278cad0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: accbd4eff69a4d44b4559891278cad0d, State: Initialized, Role: FOLLOWER
I20260810 01:47:47.444469 25704 consensus_queue.cc:260] T 00000000000000000000000000000000 P accbd4eff69a4d44b4559891278cad0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "accbd4eff69a4d44b4559891278cad0d" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 36293 } }
I20260810 01:47:47.444659 25704 raft_consensus.cc:388] T 00000000000000000000000000000000 P accbd4eff69a4d44b4559891278cad0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:47:47.444730 25704 raft_consensus.cc:482] T 00000000000000000000000000000000 P accbd4eff69a4d44b4559891278cad0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:47:47.444845 25704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P accbd4eff69a4d44b4559891278cad0d [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:47.445979 25704 raft_consensus.cc:504] T 00000000000000000000000000000000 P accbd4eff69a4d44b4559891278cad0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "accbd4eff69a4d44b4559891278cad0d" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 36293 } }
I20260810 01:47:47.446393 25704 leader_election.cc:302] T 00000000000000000000000000000000 P accbd4eff69a4d44b4559891278cad0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: accbd4eff69a4d44b4559891278cad0d; no voters:
I20260810 01:47:47.446769 25704 leader_election.cc:288] T 00000000000000000000000000000000 P accbd4eff69a4d44b4559891278cad0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:47:47.446910 25711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P accbd4eff69a4d44b4559891278cad0d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:47:47.447182 25711 raft_consensus.cc:686] T 00000000000000000000000000000000 P accbd4eff69a4d44b4559891278cad0d [term 1 LEADER]: Becoming Leader. State: Replica: accbd4eff69a4d44b4559891278cad0d, State: Running, Role: LEADER
I20260810 01:47:47.447547 25711 consensus_queue.cc:237] T 00000000000000000000000000000000 P accbd4eff69a4d44b4559891278cad0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "accbd4eff69a4d44b4559891278cad0d" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 36293 } }
I20260810 01:47:47.448414 25704 sys_catalog.cc:564] T 00000000000000000000000000000000 P accbd4eff69a4d44b4559891278cad0d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:47:47.449604 25714 sys_catalog.cc:455] T 00000000000000000000000000000000 P accbd4eff69a4d44b4559891278cad0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader accbd4eff69a4d44b4559891278cad0d. Latest consensus state: current_term: 1 leader_uuid: "accbd4eff69a4d44b4559891278cad0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "accbd4eff69a4d44b4559891278cad0d" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 36293 } } }
I20260810 01:47:47.449739 25714 sys_catalog.cc:458] T 00000000000000000000000000000000 P accbd4eff69a4d44b4559891278cad0d [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:47.450217 25713 sys_catalog.cc:455] T 00000000000000000000000000000000 P accbd4eff69a4d44b4559891278cad0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "accbd4eff69a4d44b4559891278cad0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "accbd4eff69a4d44b4559891278cad0d" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 36293 } } }
I20260810 01:47:47.450397 25713 sys_catalog.cc:458] T 00000000000000000000000000000000 P accbd4eff69a4d44b4559891278cad0d [sys.catalog]: This master's current role is: LEADER
I20260810 01:47:47.450799 25726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:47:47.455202 25726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:47:47.462265 25726 catalog_manager.cc:1347] Generated new cluster ID: 554639ba9496414b82a53eddcc85fb65
I20260810 01:47:47.462366 25726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:47:47.479449 25726 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:47:47.480988 25726 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:47:47.490561 25726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P accbd4eff69a4d44b4559891278cad0d: Generated new TSK 0
I20260810 01:47:47.491549 25726 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.77.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
I20260810 01:47:47.533743 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:0
--local_ip_for_outbound_sockets=127.9.77.129
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:36293
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:39499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:47.671176 25747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:47.671548 25747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:47.671625 25747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:47.675840 25747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:47.675911 25747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:47.675972 25747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:47:47.681595 25747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:39499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:36293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:47.683002 25747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:47.684305 25747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:47.695843 25760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:47.695868 25757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:47.695863 25764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:47.696252 25747 server_base.cc:1034] running on GCE node
I20260810 01:47:47.696751 25747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:47.697580 25747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:47.698776 25747 hybrid_clock.cc:648] HybridClock initialized: now 1786326467698731 us; error 38 us; skew 500 ppm
Aug 10 01:47:47 dist-test-slave-kvfs krb5kdc[25579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326467, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:47.705816 25747 init.cc:377] Logged in from keytab as kudu/127.9.77.129@KRBTEST.COM (short username kudu)
I20260810 01:47:47.706940 25747 webserver.cc:460] Webserver started at http://127.9.77.129:39653/ using document root <none> and password file <none>
I20260810 01:47:47.707643 25747 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:47.707710 25747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:47.707933 25747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:47.710530 25747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/data/instance:
uuid: "67cf84cdf69a49e3bee56787823a2056"
format_stamp: "Formatted at 2026-08-10 01:47:47 on dist-test-slave-kvfs"
server_key: "cba605ee859deaf348836b5b0462f9ed"
server_key_iv: "b2f0ea314361c75917efad6969420215"
server_key_version: "encryptionkey@0"
I20260810 01:47:47.711160 25747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance:
uuid: "67cf84cdf69a49e3bee56787823a2056"
format_stamp: "Formatted at 2026-08-10 01:47:47 on dist-test-slave-kvfs"
server_key: "cba605ee859deaf348836b5b0462f9ed"
server_key_iv: "b2f0ea314361c75917efad6969420215"
server_key_version: "encryptionkey@0"
I20260810 01:47:47.715863 25747 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 01:47:47.718932 25776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:47.720564 25747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:47:47.720726 25747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "67cf84cdf69a49e3bee56787823a2056"
format_stamp: "Formatted at 2026-08-10 01:47:47 on dist-test-slave-kvfs"
server_key: "cba605ee859deaf348836b5b0462f9ed"
server_key_iv: "b2f0ea314361c75917efad6969420215"
server_key_version: "encryptionkey@0"
I20260810 01:47:47.720888 25747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:47.739785 25747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:47.743858 25747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:47.744151 25747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:47.744939 25747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:47.745990 25747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:47.746052 25747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:47.746124 25747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:47.746155 25747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:47.757217 25747 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:35683
I20260810 01:47:47.757239 25948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:35683 every 8 connection(s)
I20260810 01:47:47.758345 25747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:47:47.763747 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 25747
I20260810 01:47:47.763921 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance
I20260810 01:47:47.764295 9526 external_mini_cluster.cc:1428] Setting key e18c2fc4afb7c0d962a941712e48d3c7
Aug 10 01:47:47 dist-test-slave-kvfs krb5kdc[25579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326467, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:47:47.776054 25709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:47.760616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:54585 (local address 127.9.77.190:36293)
0810 01:47:47.760802 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:47.760807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:47.761847 (+ 1040us) server_negotiation.cc:407] Connection header received
0810 01:47:47.763407 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:47.763414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:47.763492 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:47.763624 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:47.765239 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:47.766073 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:47.767989 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:47.768371 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:47.768474 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:47.771754 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:47.771758 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:47.771760 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:47.771845 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:47.773579 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:47.774158 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:47.774162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:47.774164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:47.774258 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:47.774672 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:47.774675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:47.774676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:47.774902 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:47.775043 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:47.775662 (+ 619us) server_negotiation.cc:299] Negotiation successful
0810 01:47:47.775818 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 01:47:47.777150 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:47.761146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:36293 (local address 127.9.77.129:54585)
0810 01:47:47.761662 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:47.761713 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:47.763146 (+ 1433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:47.763804 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:47.763812 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:47.764343 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:47.765064 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:47.765086 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:47.766233 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:47.766238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:47.767833 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:47.767849 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:47.768551 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:47.768558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:47.768834 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:47.769630 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:47.769649 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:47.771536 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:47.773710 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:47.773718 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:47.773739 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:47.774044 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:47.774367 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:47.774370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:47.774372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:47.774559 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:47.775041 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:47.775050 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:47.775369 (+ 319us) client_negotiation.cc:769] Sending connection context
0810 01:47:47.775609 (+ 240us) client_negotiation.cc:240] Negotiation successful
0810 01:47:47.776059 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":136,"threads_started":1}
WARNING: no policy specified for kudu/127.9.77.130@KRBTEST.COM; defaulting to no policy
I20260810 01:47:47.778736 25951 heartbeater.cc:347] Connected to a master server at 127.9.77.190:36293
I20260810 01:47:47.779067 25951 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:47.779695 25951 heartbeater.cc:511] Master 127.9.77.190:36293 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.130@KRBTEST.COM" created.
I20260810 01:47:47.781829 25646 ts_manager.cc:194] Registered new tserver with Master: 67cf84cdf69a49e3bee56787823a2056 (127.9.77.129:35683)
I20260810 01:47:47.783202 25646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.129@KRBTEST.COM'} at 127.9.77.129:54585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
I20260810 01:47:47.799993 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:0
--local_ip_for_outbound_sockets=127.9.77.130
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:36293
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:39499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:47.945760 25959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:47.946295 25959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:47.946388 25959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:47.956369 25959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:47.956502 25959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:47.956655 25959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:47:47.967144 25959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:39499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:36293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:47.969297 25959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:47.971179 25959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:47.985419 25971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:47.985425 25975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:47.985423 25970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:47.985879 25959 server_base.cc:1034] running on GCE node
I20260810 01:47:47.986476 25959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:47.987270 25959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:47.988445 25959 hybrid_clock.cc:648] HybridClock initialized: now 1786326467988406 us; error 61 us; skew 500 ppm
Aug 10 01:47:47 dist-test-slave-kvfs krb5kdc[25579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326467, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:47.992265 25959 init.cc:377] Logged in from keytab as kudu/127.9.77.130@KRBTEST.COM (short username kudu)
I20260810 01:47:47.993453 25959 webserver.cc:460] Webserver started at http://127.9.77.130:38337/ using document root <none> and password file <none>
I20260810 01:47:47.994136 25959 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:47.994201 25959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:47.994421 25959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:47.997002 25959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/data/instance:
uuid: "4c7643b3b0f447319e724bab8a97140f"
format_stamp: "Formatted at 2026-08-10 01:47:47 on dist-test-slave-kvfs"
server_key: "db6c2832aa56c400cf1906a02f7fdafc"
server_key_iv: "69c91d2f64414e5fa251a37a8b8f63ba"
server_key_version: "encryptionkey@0"
I20260810 01:47:47.997627 25959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance:
uuid: "4c7643b3b0f447319e724bab8a97140f"
format_stamp: "Formatted at 2026-08-10 01:47:47 on dist-test-slave-kvfs"
server_key: "db6c2832aa56c400cf1906a02f7fdafc"
server_key_iv: "69c91d2f64414e5fa251a37a8b8f63ba"
server_key_version: "encryptionkey@0"
I20260810 01:47:48.002180 25959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 01:47:48.005187 25982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:48.006886 25959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:47:48.007063 25959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "4c7643b3b0f447319e724bab8a97140f"
format_stamp: "Formatted at 2026-08-10 01:47:47 on dist-test-slave-kvfs"
server_key: "db6c2832aa56c400cf1906a02f7fdafc"
server_key_iv: "69c91d2f64414e5fa251a37a8b8f63ba"
server_key_version: "encryptionkey@0"
I20260810 01:47:48.007207 25959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:48.023578 25959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:48.027549 25959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:48.027807 25959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:48.028630 25959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:48.029671 25959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:48.029748 25959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:48.029855 25959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:48.029903 25959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:48.040506 25959 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:41167
I20260810 01:47:48.040529 26158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:41167 every 8 connection(s)
I20260810 01:47:48.041742 25959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:47:48.049477 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 25959
I20260810 01:47:48.049664 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance
I20260810 01:47:48.050115 9526 external_mini_cluster.cc:1428] Setting key f1460218807cee2ae5332c8a0555f0d6
Aug 10 01:47:48 dist-test-slave-kvfs krb5kdc[25579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326467, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:47:48.060350 25709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:48.044176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:53923 (local address 127.9.77.190:36293)
0810 01:47:48.044381 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:48.044387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:48.045435 (+ 1048us) server_negotiation.cc:407] Connection header received
0810 01:47:48.047020 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:48.047024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:48.047089 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:48.047202 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:48.048905 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.049797 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.051491 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.051866 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.051996 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:48.055433 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:48.055440 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:48.055445 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:48.055486 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:48.057581 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:48.058233 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:48.058240 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:48.058245 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:48.058358 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:48.058791 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:48.058797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:48.058808 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:48.059118 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:48.059243 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:48.059932 (+ 689us) server_negotiation.cc:299] Negotiation successful
0810 01:47:48.060127 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 01:47:48.061484 26162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:48.044637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:36293 (local address 127.9.77.130:53923)
0810 01:47:48.045225 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:48.045283 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:48.046746 (+ 1463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:48.047456 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:48.047464 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:48.047914 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:48.048705 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:48.048725 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.049948 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.049953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:48.051354 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:48.051363 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.052090 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.052095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:48.052344 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:48.053127 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:48.053157 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:48.055221 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:48.057723 (+ 2502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:48.057730 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:48.057774 (+ 44us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:48.058106 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:48.058488 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:48.058492 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:48.058493 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:48.058672 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:48.059257 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:48.059264 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:48.059592 (+ 328us) client_negotiation.cc:769] Sending connection context
0810 01:47:48.059856 (+ 264us) client_negotiation.cc:240] Negotiation successful
0810 01:47:48.060322 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":140,"threads_started":1}
I20260810 01:47:48.062991 26160 heartbeater.cc:347] Connected to a master server at 127.9.77.190:36293
I20260810 01:47:48.063319 26160 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:48.063952 26160 heartbeater.cc:511] Master 127.9.77.190:36293 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.77.131@KRBTEST.COM; defaulting to no policy
I20260810 01:47:48.065429 25646 ts_manager.cc:194] Registered new tserver with Master: 4c7643b3b0f447319e724bab8a97140f (127.9.77.130:41167)
I20260810 01:47:48.066174 25646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.130@KRBTEST.COM'} at 127.9.77.130:53923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
I20260810 01:47:48.087504 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:0
--local_ip_for_outbound_sockets=127.9.77.131
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:36293
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:39499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:47:48.225426 26166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:47:48.225868 26166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:47:48.225950 26166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:47:48.230448 26166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:47:48.230584 26166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:47:48.230695 26166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:47:48.237164 26166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:39499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:36293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:47:48.238831 26166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:47:48.240294 26166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:47:48.252884 26175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:48.252887 26179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:47:48.252887 26176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:47:48.253193 26166 server_base.cc:1034] running on GCE node
I20260810 01:47:48.253772 26166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:47:48.254614 26166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:47:48.255906 26166 hybrid_clock.cc:648] HybridClock initialized: now 1786326468255851 us; error 59 us; skew 500 ppm
Aug 10 01:47:48 dist-test-slave-kvfs krb5kdc[25579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326468, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:47:48.259728 26166 init.cc:377] Logged in from keytab as kudu/127.9.77.131@KRBTEST.COM (short username kudu)
I20260810 01:47:48.261149 26166 webserver.cc:460] Webserver started at http://127.9.77.131:38653/ using document root <none> and password file <none>
I20260810 01:47:48.261955 26166 fs_manager.cc:359] Metadata directory not provided
I20260810 01:47:48.262074 26166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:47:48.262385 26166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:47:48.265225 26166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/data/instance:
uuid: "8d9a4167a9f744f1b18c735265a626ac"
format_stamp: "Formatted at 2026-08-10 01:47:48 on dist-test-slave-kvfs"
server_key: "d4f0aad272973ba05c915e55e5cab97d"
server_key_iv: "8289a1e28d7896c5173c4259e5e2b227"
server_key_version: "encryptionkey@0"
I20260810 01:47:48.265972 26166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance:
uuid: "8d9a4167a9f744f1b18c735265a626ac"
format_stamp: "Formatted at 2026-08-10 01:47:48 on dist-test-slave-kvfs"
server_key: "d4f0aad272973ba05c915e55e5cab97d"
server_key_iv: "8289a1e28d7896c5173c4259e5e2b227"
server_key_version: "encryptionkey@0"
I20260810 01:47:48.271076 26166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 01:47:48.274410 26187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:48.276047 26166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 01:47:48.276237 26166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "8d9a4167a9f744f1b18c735265a626ac"
format_stamp: "Formatted at 2026-08-10 01:47:48 on dist-test-slave-kvfs"
server_key: "d4f0aad272973ba05c915e55e5cab97d"
server_key_iv: "8289a1e28d7896c5173c4259e5e2b227"
server_key_version: "encryptionkey@0"
I20260810 01:47:48.276412 26166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:47:48.321213 26166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:47:48.325157 26166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:47:48.325421 26166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:47:48.326288 26166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:47:48.327410 26166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:47:48.327499 26166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:48.327582 26166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:47:48.327638 26166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:47:48.338104 26166 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:42261
I20260810 01:47:48.338135 26368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:42261 every 8 connection(s)
I20260810 01:47:48.339238 26166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:47:48.347724 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 26166
I20260810 01:47:48.347883 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance
I20260810 01:47:48.348330 9526 external_mini_cluster.cc:1428] Setting key feda80f858bd118a76bb747fcfe09357
Aug 10 01:47:48 dist-test-slave-kvfs krb5kdc[25579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326468, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:47:48.357457 25709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:48.341562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:55101 (local address 127.9.77.190:36293)
0810 01:47:48.341757 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:48.341762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:48.342989 (+ 1227us) server_negotiation.cc:407] Connection header received
0810 01:47:48.344613 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:48.344617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:48.344690 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:48.344786 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:48.346600 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.347630 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.349571 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.349844 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.350002 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:48.353207 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:48.353210 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:48.353213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:48.353251 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:48.354928 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:48.355562 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:48.355566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:48.355568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:48.355670 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:48.356108 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:48.356111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:48.356113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:48.356356 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:48.356472 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:48.357094 (+ 622us) server_negotiation.cc:299] Negotiation successful
0810 01:47:48.357260 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 01:47:48.358542 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:48.342218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:36293 (local address 127.9.77.131:55101)
0810 01:47:48.342799 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:48.342852 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:48.344343 (+ 1491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:48.345014 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:48.345026 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:48.345595 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:47:48.346411 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:48.346438 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.347803 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.347807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:48.349444 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:48.349454 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.350054 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.350058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:48.350288 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:48.351093 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:47:48.351112 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:47:48.352986 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:47:48.355099 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:48.355106 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:48.355127 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:48.355442 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:48.355784 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:47:48.355789 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:47:48.355792 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:47:48.355964 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:47:48.356479 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:47:48.356489 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:47:48.356814 (+ 325us) client_negotiation.cc:769] Sending connection context
0810 01:47:48.357036 (+ 222us) client_negotiation.cc:240] Negotiation successful
0810 01:47:48.357473 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":153,"threads_started":1}
I20260810 01:47:48.359974 26369 heartbeater.cc:347] Connected to a master server at 127.9.77.190:36293
I20260810 01:47:48.360355 26369 heartbeater.cc:464] Registering TS with master...
I20260810 01:47:48.361047 26369 heartbeater.cc:511] Master 127.9.77.190:36293 requested a full tablet report, sending...
I20260810 01:47:48.362397 25646 ts_manager.cc:194] Registered new tserver with Master: 8d9a4167a9f744f1b18c735265a626ac (127.9.77.131:42261)
I20260810 01:47:48.363111 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 01:47:48.363206 25646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.131@KRBTEST.COM'} at 127.9.77.131:55101
I20260810 01:47:48.374354 25709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:48.365923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49476 (local address 127.9.77.190:36293)
0810 01:47:48.366069 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:48.366074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:48.366269 (+ 195us) server_negotiation.cc:407] Connection header received
0810 01:47:48.366365 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:48.366367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:48.366411 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:48.366489 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:47:48.367267 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.367899 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.368883 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.369150 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.369235 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:48.370740 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:47:48.370744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:47:48.370746 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:47:48.370770 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:47:48.372599 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:48.373062 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:48.373066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:48.373067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:48.373165 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:47:48.373458 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:47:48.373461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:47:48.373463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:47:48.373750 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:47:48.373892 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:48.374088 (+ 196us) server_negotiation.cc:299] Negotiation successful
0810 01:47:48.374208 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 01:47:48.377915 25646 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 01:47:48.381845 25646 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:47:48.399206 26385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:48.392625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51136 (local address 127.9.77.131:42261)
0810 01:47:48.393731 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:48.393744 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:48.393768 (+ 24us) server_negotiation.cc:407] Connection header received
0810 01:47:48.393864 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:48.393870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:48.394113 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:48.394298 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:48.394866 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.396171 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.397734 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.398610 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.398745 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:48.398911 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:48.398969 (+ 58us) server_negotiation.cc:299] Negotiation successful
0810 01:47:48.399066 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":992,"thread_start_us":141,"threads_started":1}
I20260810 01:47:48.399173 26382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:48.392152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:42261 (local address 127.0.0.1:51136)
0810 01:47:48.393145 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:48.393210 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:48.393396 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:48.394299 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:48.394307 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:48.394349 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:48.394591 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:48.394598 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.396320 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.396324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:48.397595 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:48.397606 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.398743 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.398747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:48.398839 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:48.398862 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 01:47:48.398927 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 01:47:48.399001 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"mutex_wait_us":155,"spinlock_wait_cycles":15744,"thread_start_us":100,"threads_started":1}
I20260810 01:47:48.399176 26390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:48.393013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:41167 (local address 127.0.0.1:38964)
0810 01:47:48.393434 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:48.393450 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:48.393559 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:48.394458 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:48.394461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:48.394477 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:48.394743 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:48.394748 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.396364 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.396366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:48.397317 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:48.397326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.398540 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.398547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:48.398670 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:48.398712 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 01:47:48.398815 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 01:47:48.398970 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":247,"thread_start_us":119,"threads_started":1}
I20260810 01:47:48.399225 26384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:48.392107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:35683 (local address 127.0.0.1:55538)
0810 01:47:48.393145 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:48.393210 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:48.393396 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:48.394300 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:48.394306 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:48.394339 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:48.394570 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:48.394580 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.395943 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.395949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:48.397551 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:48.397562 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.398865 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.398868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:48.398954 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:48.398974 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 01:47:48.399044 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 01:47:48.399109 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":48,"spinlock_wait_cycles":10112,"thread_start_us":113,"threads_started":1}
I20260810 01:47:48.399832 26389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:48.392982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55538 (local address 127.9.77.129:35683)
0810 01:47:48.393731 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:48.393742 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:48.393781 (+ 39us) server_negotiation.cc:407] Connection header received
0810 01:47:48.393864 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:48.393870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:48.394113 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:48.394541 (+ 428us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:48.394892 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.395817 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.397703 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.398497 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.398600 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:48.398786 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:48.399512 (+ 726us) server_negotiation.cc:299] Negotiation successful
0810 01:47:48.399652 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":127,"threads_started":1}
I20260810 01:47:48.400964 26391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:48.393564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38964 (local address 127.9.77.130:41167)
0810 01:47:48.393973 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:48.393986 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:48.394019 (+ 33us) server_negotiation.cc:407] Connection header received
0810 01:47:48.394096 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:48.394100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:48.394291 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:48.394610 (+ 319us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:48.394828 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.396198 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.397442 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.398380 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.399273 (+ 893us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:48.400624 (+ 1351us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:48.400687 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 01:47:48.400794 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":175,"threads_started":1}
I20260810 01:47:48.402842 26258 tablet_service.cc:1467] Processing CreateTablet for tablet ec35cd2b06534c46bc23516cfca3defe (DEFAULT_TABLE table=test-table [id=2acc34b9159e4a7e88517b482fdb8960]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:47:48.402846 25840 tablet_service.cc:1467] Processing CreateTablet for tablet ec35cd2b06534c46bc23516cfca3defe (DEFAULT_TABLE table=test-table [id=2acc34b9159e4a7e88517b482fdb8960]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:47:48.403602 26056 tablet_service.cc:1467] Processing CreateTablet for tablet ec35cd2b06534c46bc23516cfca3defe (DEFAULT_TABLE table=test-table [id=2acc34b9159e4a7e88517b482fdb8960]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:47:48.404186 25840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec35cd2b06534c46bc23516cfca3defe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:48.404371 26258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec35cd2b06534c46bc23516cfca3defe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:48.404590 26056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec35cd2b06534c46bc23516cfca3defe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:47:48.411957 26392 tablet_bootstrap.cc:492] T ec35cd2b06534c46bc23516cfca3defe P 8d9a4167a9f744f1b18c735265a626ac: Bootstrap starting.
I20260810 01:47:48.413357 26394 tablet_bootstrap.cc:492] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f: Bootstrap starting.
I20260810 01:47:48.415475 26392 tablet_bootstrap.cc:654] T ec35cd2b06534c46bc23516cfca3defe P 8d9a4167a9f744f1b18c735265a626ac: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:48.416110 26394 tablet_bootstrap.cc:654] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:48.416457 26393 tablet_bootstrap.cc:492] T ec35cd2b06534c46bc23516cfca3defe P 67cf84cdf69a49e3bee56787823a2056: Bootstrap starting.
I20260810 01:47:48.416530 26392 log.cc:824] T ec35cd2b06534c46bc23516cfca3defe P 8d9a4167a9f744f1b18c735265a626ac: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:48.417050 26394 log.cc:824] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:48.418931 26393 tablet_bootstrap.cc:654] T ec35cd2b06534c46bc23516cfca3defe P 67cf84cdf69a49e3bee56787823a2056: Neither blocks nor log segments found. Creating new log.
I20260810 01:47:48.419212 26394 tablet_bootstrap.cc:492] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f: No bootstrap required, opened a new log
I20260810 01:47:48.419211 26392 tablet_bootstrap.cc:492] T ec35cd2b06534c46bc23516cfca3defe P 8d9a4167a9f744f1b18c735265a626ac: No bootstrap required, opened a new log
I20260810 01:47:48.419466 26392 ts_tablet_manager.cc:1397] T ec35cd2b06534c46bc23516cfca3defe P 8d9a4167a9f744f1b18c735265a626ac: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260810 01:47:48.419466 26394 ts_tablet_manager.cc:1397] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 01:47:48.419811 26393 log.cc:824] T ec35cd2b06534c46bc23516cfca3defe P 67cf84cdf69a49e3bee56787823a2056: Log is configured to *not* fsync() on all Append() calls
I20260810 01:47:48.422012 26393 tablet_bootstrap.cc:492] T ec35cd2b06534c46bc23516cfca3defe P 67cf84cdf69a49e3bee56787823a2056: No bootstrap required, opened a new log
I20260810 01:47:48.422231 26393 ts_tablet_manager.cc:1397] T ec35cd2b06534c46bc23516cfca3defe P 67cf84cdf69a49e3bee56787823a2056: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 01:47:48.424162 26392 raft_consensus.cc:348] T ec35cd2b06534c46bc23516cfca3defe P 8d9a4167a9f744f1b18c735265a626ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf84cdf69a49e3bee56787823a2056" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 35683 } } peers { permanent_uuid: "8d9a4167a9f744f1b18c735265a626ac" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 42261 } } peers { permanent_uuid: "4c7643b3b0f447319e724bab8a97140f" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 41167 } }
I20260810 01:47:48.424667 26392 raft_consensus.cc:374] T ec35cd2b06534c46bc23516cfca3defe P 8d9a4167a9f744f1b18c735265a626ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:48.424789 26392 raft_consensus.cc:729] T ec35cd2b06534c46bc23516cfca3defe P 8d9a4167a9f744f1b18c735265a626ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d9a4167a9f744f1b18c735265a626ac, State: Initialized, Role: FOLLOWER
I20260810 01:47:48.424980 26394 raft_consensus.cc:348] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf84cdf69a49e3bee56787823a2056" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 35683 } } peers { permanent_uuid: "8d9a4167a9f744f1b18c735265a626ac" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 42261 } } peers { permanent_uuid: "4c7643b3b0f447319e724bab8a97140f" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 41167 } }
I20260810 01:47:48.425275 26394 raft_consensus.cc:374] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:48.425338 26394 raft_consensus.cc:729] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c7643b3b0f447319e724bab8a97140f, State: Initialized, Role: FOLLOWER
I20260810 01:47:48.425726 26394 consensus_queue.cc:260] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf84cdf69a49e3bee56787823a2056" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 35683 } } peers { permanent_uuid: "8d9a4167a9f744f1b18c735265a626ac" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 42261 } } peers { permanent_uuid: "4c7643b3b0f447319e724bab8a97140f" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 41167 } }
I20260810 01:47:48.425716 26392 consensus_queue.cc:260] T ec35cd2b06534c46bc23516cfca3defe P 8d9a4167a9f744f1b18c735265a626ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf84cdf69a49e3bee56787823a2056" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 35683 } } peers { permanent_uuid: "8d9a4167a9f744f1b18c735265a626ac" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 42261 } } peers { permanent_uuid: "4c7643b3b0f447319e724bab8a97140f" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 41167 } }
I20260810 01:47:48.427004 26394 ts_tablet_manager.cc:1428] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 01:47:48.427004 26392 ts_tablet_manager.cc:1428] T ec35cd2b06534c46bc23516cfca3defe P 8d9a4167a9f744f1b18c735265a626ac: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 01:47:48.427485 26369 heartbeater.cc:503] Master 127.9.77.190:36293 was elected leader, sending a full tablet report...
I20260810 01:47:48.427490 26160 heartbeater.cc:503] Master 127.9.77.190:36293 was elected leader, sending a full tablet report...
I20260810 01:47:48.427392 26393 raft_consensus.cc:348] T ec35cd2b06534c46bc23516cfca3defe P 67cf84cdf69a49e3bee56787823a2056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf84cdf69a49e3bee56787823a2056" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 35683 } } peers { permanent_uuid: "8d9a4167a9f744f1b18c735265a626ac" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 42261 } } peers { permanent_uuid: "4c7643b3b0f447319e724bab8a97140f" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 41167 } }
I20260810 01:47:48.427716 26393 raft_consensus.cc:374] T ec35cd2b06534c46bc23516cfca3defe P 67cf84cdf69a49e3bee56787823a2056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:47:48.427778 26393 raft_consensus.cc:729] T ec35cd2b06534c46bc23516cfca3defe P 67cf84cdf69a49e3bee56787823a2056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67cf84cdf69a49e3bee56787823a2056, State: Initialized, Role: FOLLOWER
I20260810 01:47:48.428279 26393 consensus_queue.cc:260] T ec35cd2b06534c46bc23516cfca3defe P 67cf84cdf69a49e3bee56787823a2056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf84cdf69a49e3bee56787823a2056" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 35683 } } peers { permanent_uuid: "8d9a4167a9f744f1b18c735265a626ac" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 42261 } } peers { permanent_uuid: "4c7643b3b0f447319e724bab8a97140f" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 41167 } }
I20260810 01:47:48.429445 26393 ts_tablet_manager.cc:1428] T ec35cd2b06534c46bc23516cfca3defe P 67cf84cdf69a49e3bee56787823a2056: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 01:47:48.429824 25951 heartbeater.cc:503] Master 127.9.77.190:36293 was elected leader, sending a full tablet report...
W20260810 01:47:48.510070 25952 tablet.cc:2369] T ec35cd2b06534c46bc23516cfca3defe P 67cf84cdf69a49e3bee56787823a2056: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 01:47:48.543215 26161 tablet.cc:2369] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 01:47:48.590437 26370 tablet.cc:2369] T ec35cd2b06534c46bc23516cfca3defe P 8d9a4167a9f744f1b18c735265a626ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 01:47:48.632895 26399 raft_consensus.cc:482] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:47:48.633180 26399 raft_consensus.cc:504] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf84cdf69a49e3bee56787823a2056" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 35683 } } peers { permanent_uuid: "8d9a4167a9f744f1b18c735265a626ac" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 42261 } } peers { permanent_uuid: "4c7643b3b0f447319e724bab8a97140f" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 41167 } }
I20260810 01:47:48.634697 26399 leader_election.cc:288] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67cf84cdf69a49e3bee56787823a2056 (127.9.77.129:35683), 8d9a4167a9f744f1b18c735265a626ac (127.9.77.131:42261)
I20260810 01:47:48.639504 26385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:48.635344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:42349 (local address 127.9.77.131:42261)
0810 01:47:48.635553 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:48.635561 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:48.635724 (+ 163us) server_negotiation.cc:407] Connection header received
0810 01:47:48.635883 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:48.635886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:48.635938 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:48.636066 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:48.636410 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.637016 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.638294 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.638783 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.638935 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:48.639001 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:48.639228 (+ 227us) server_negotiation.cc:299] Negotiation successful
0810 01:47:48.639327 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260810 01:47:48.639504 26389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:48.635242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:50959 (local address 127.9.77.129:35683)
0810 01:47:48.635466 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:48.635473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:48.635636 (+ 163us) server_negotiation.cc:407] Connection header received
0810 01:47:48.635859 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:48.635863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:48.635928 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:48.635998 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:47:48.636446 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.636984 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.638320 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.638775 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.638915 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:48.638996 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:48.639230 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 01:47:48.639332 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260810 01:47:48.639526 26403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:48.635048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:42261 (local address 127.9.77.130:42349)
0810 01:47:48.635583 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:48.635619 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:48.635736 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:48.636078 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:48.636081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:48.636109 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:48.636276 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:48.636282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.637158 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.637160 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:48.638151 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:48.638160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.638934 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.638937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:48.639066 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:48.639107 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 01:47:48.639204 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 01:47:48.639328 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":167,"spinlock_wait_cycles":2304,"thread_start_us":124,"threads_started":1}
I20260810 01:47:48.639530 26406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:48.635048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:35683 (local address 127.9.77.130:50959)
0810 01:47:48.635489 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:48.635508 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 01:47:48.635704 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:47:48.636071 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:47:48.636075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:47:48.636092 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:47:48.636294 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:48.636300 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.637099 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.637106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:48.638153 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:47:48.638163 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.638911 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.638915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:47:48.639043 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:48.639111 (+ 68us) client_negotiation.cc:769] Sending connection context
0810 01:47:48.639212 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 01:47:48.639327 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":94,"spinlock_wait_cycles":12672,"thread_start_us":98,"threads_started":1}
I20260810 01:47:48.640703 25878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec35cd2b06534c46bc23516cfca3defe" candidate_uuid: "4c7643b3b0f447319e724bab8a97140f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67cf84cdf69a49e3bee56787823a2056" is_pre_election: true
I20260810 01:47:48.640662 26287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec35cd2b06534c46bc23516cfca3defe" candidate_uuid: "4c7643b3b0f447319e724bab8a97140f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d9a4167a9f744f1b18c735265a626ac" is_pre_election: true
I20260810 01:47:48.641125 25878 raft_consensus.cc:2454] T ec35cd2b06534c46bc23516cfca3defe P 67cf84cdf69a49e3bee56787823a2056 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c7643b3b0f447319e724bab8a97140f in term 0.
I20260810 01:47:48.641126 26287 raft_consensus.cc:2454] T ec35cd2b06534c46bc23516cfca3defe P 8d9a4167a9f744f1b18c735265a626ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c7643b3b0f447319e724bab8a97140f in term 0.
I20260810 01:47:48.641747 25983 leader_election.cc:302] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4c7643b3b0f447319e724bab8a97140f, 67cf84cdf69a49e3bee56787823a2056, 8d9a4167a9f744f1b18c735265a626ac; no voters:
I20260810 01:47:48.642110 26399 raft_consensus.cc:2781] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:47:48.642226 26399 raft_consensus.cc:482] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:47:48.642321 26399 raft_consensus.cc:3037] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:48.643858 26399 raft_consensus.cc:504] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf84cdf69a49e3bee56787823a2056" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 35683 } } peers { permanent_uuid: "8d9a4167a9f744f1b18c735265a626ac" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 42261 } } peers { permanent_uuid: "4c7643b3b0f447319e724bab8a97140f" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 41167 } }
I20260810 01:47:48.644321 26399 leader_election.cc:288] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f [CANDIDATE]: Term 1 election: Requested vote from peers 67cf84cdf69a49e3bee56787823a2056 (127.9.77.129:35683), 8d9a4167a9f744f1b18c735265a626ac (127.9.77.131:42261)
I20260810 01:47:48.644742 25878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec35cd2b06534c46bc23516cfca3defe" candidate_uuid: "4c7643b3b0f447319e724bab8a97140f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67cf84cdf69a49e3bee56787823a2056"
I20260810 01:47:48.644742 26287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec35cd2b06534c46bc23516cfca3defe" candidate_uuid: "4c7643b3b0f447319e724bab8a97140f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d9a4167a9f744f1b18c735265a626ac"
I20260810 01:47:48.644897 26287 raft_consensus.cc:3037] T ec35cd2b06534c46bc23516cfca3defe P 8d9a4167a9f744f1b18c735265a626ac [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:48.644896 25878 raft_consensus.cc:3037] T ec35cd2b06534c46bc23516cfca3defe P 67cf84cdf69a49e3bee56787823a2056 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:47:48.646479 26287 raft_consensus.cc:2454] T ec35cd2b06534c46bc23516cfca3defe P 8d9a4167a9f744f1b18c735265a626ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c7643b3b0f447319e724bab8a97140f in term 1.
I20260810 01:47:48.646479 25878 raft_consensus.cc:2454] T ec35cd2b06534c46bc23516cfca3defe P 67cf84cdf69a49e3bee56787823a2056 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c7643b3b0f447319e724bab8a97140f in term 1.
I20260810 01:47:48.646795 25983 leader_election.cc:302] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c7643b3b0f447319e724bab8a97140f, 67cf84cdf69a49e3bee56787823a2056; no voters:
I20260810 01:47:48.646984 26399 raft_consensus.cc:2781] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:47:48.647246 26399 raft_consensus.cc:686] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f [term 1 LEADER]: Becoming Leader. State: Replica: 4c7643b3b0f447319e724bab8a97140f, State: Running, Role: LEADER
I20260810 01:47:48.647678 26399 consensus_queue.cc:237] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf84cdf69a49e3bee56787823a2056" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 35683 } } peers { permanent_uuid: "8d9a4167a9f744f1b18c735265a626ac" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 42261 } } peers { permanent_uuid: "4c7643b3b0f447319e724bab8a97140f" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 41167 } }
I20260810 01:47:48.652011 25645 catalog_manager.cc:5495] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c7643b3b0f447319e724bab8a97140f (127.9.77.130). New cstate: current_term: 1 leader_uuid: "4c7643b3b0f447319e724bab8a97140f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cf84cdf69a49e3bee56787823a2056" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 35683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d9a4167a9f744f1b18c735265a626ac" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 42261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c7643b3b0f447319e724bab8a97140f" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 41167 } health_report { overall_health: HEALTHY } } }
I20260810 01:47:48.697685 26391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:48.692767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38972 (local address 127.9.77.130:41167)
0810 01:47:48.692936 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 01:47:48.692940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:47:48.692957 (+ 17us) server_negotiation.cc:407] Connection header received
0810 01:47:48.693068 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:47:48.693070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:47:48.693153 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:47:48.693287 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:47:48.693642 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.694288 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.695543 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:47:48.695808 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:47:48.695942 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:47:48.696589 (+ 647us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:47:48.697226 (+ 637us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:47:48.697343 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 01:47:48.697405 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 01:47:48.697521 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 01:47:48.705569 26287 raft_consensus.cc:1261] T ec35cd2b06534c46bc23516cfca3defe P 8d9a4167a9f744f1b18c735265a626ac [term 1 FOLLOWER]: Refusing update from remote peer 4c7643b3b0f447319e724bab8a97140f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:47:48.705569 25878 raft_consensus.cc:1261] T ec35cd2b06534c46bc23516cfca3defe P 67cf84cdf69a49e3bee56787823a2056 [term 1 FOLLOWER]: Refusing update from remote peer 4c7643b3b0f447319e724bab8a97140f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:47:48.706588 26399 consensus_queue.cc:1035] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d9a4167a9f744f1b18c735265a626ac" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 42261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:47:48.706877 26407 consensus_queue.cc:1035] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f [LEADER]: Connected to new peer: Peer: permanent_uuid: "67cf84cdf69a49e3bee56787823a2056" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 35683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:47:48.728412 25643 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:47:48.728700 25643 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:47:48.731992 25643 catalog_manager.cc:5782] T 00000000000000000000000000000000 P accbd4eff69a4d44b4559891278cad0d: Sending DeleteTablet for 3 replicas of tablet ec35cd2b06534c46bc23516cfca3defe
I20260810 01:47:48.733249 26056 tablet_service.cc:1514] Processing DeleteTablet for tablet ec35cd2b06534c46bc23516cfca3defe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:47:48 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:38964
I20260810 01:47:48.733328 26258 tablet_service.cc:1514] Processing DeleteTablet for tablet ec35cd2b06534c46bc23516cfca3defe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:47:48 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:51136
I20260810 01:47:48.734023 26424 tablet_replica.cc:330] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f: stopping tablet replica
I20260810 01:47:48.734232 25840 tablet_service.cc:1514] Processing DeleteTablet for tablet ec35cd2b06534c46bc23516cfca3defe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:47:48 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:55538
I20260810 01:47:48.734498 26426 tablet_replica.cc:330] T ec35cd2b06534c46bc23516cfca3defe P 8d9a4167a9f744f1b18c735265a626ac: stopping tablet replica
I20260810 01:47:48.734931 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 25747
I20260810 01:47:48.735409 26424 raft_consensus.cc:2229] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f [term 1 LEADER]: Raft consensus shutting down.
I20260810 01:47:48.735661 26426 raft_consensus.cc:2229] T ec35cd2b06534c46bc23516cfca3defe P 8d9a4167a9f744f1b18c735265a626ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:47:48.736397 26424 raft_consensus.cc:2258] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:47:48.737241 26426 raft_consensus.cc:2258] T ec35cd2b06534c46bc23516cfca3defe P 8d9a4167a9f744f1b18c735265a626ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:47:48.741209 26426 ts_tablet_manager.cc:1905] T ec35cd2b06534c46bc23516cfca3defe P 8d9a4167a9f744f1b18c735265a626ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:47:48.741226 26424 ts_tablet_manager.cc:1905] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:47:48.745518 26424 ts_tablet_manager.cc:1918] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:47:48.745792 26424 log.cc:1196] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-1/wal/wals/ec35cd2b06534c46bc23516cfca3defe
I20260810 01:47:48.745924 26426 ts_tablet_manager.cc:1918] T ec35cd2b06534c46bc23516cfca3defe P 8d9a4167a9f744f1b18c735265a626ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:47:48.746065 26426 log.cc:1196] T ec35cd2b06534c46bc23516cfca3defe P 8d9a4167a9f744f1b18c735265a626ac: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786326392877349-9526-0/minicluster-data/ts-2/wal/wals/ec35cd2b06534c46bc23516cfca3defe
I20260810 01:47:48.746381 26424 ts_tablet_manager.cc:1939] T ec35cd2b06534c46bc23516cfca3defe P 4c7643b3b0f447319e724bab8a97140f: Deleting consensus metadata
I20260810 01:47:48.746542 26426 ts_tablet_manager.cc:1939] T ec35cd2b06534c46bc23516cfca3defe P 8d9a4167a9f744f1b18c735265a626ac: Deleting consensus metadata
I20260810 01:47:48.748231 25622 catalog_manager.cc:4841] TS 4c7643b3b0f447319e724bab8a97140f (127.9.77.130:41167): tablet ec35cd2b06534c46bc23516cfca3defe (table test-table [id=2acc34b9159e4a7e88517b482fdb8960]) successfully deleted
I20260810 01:47:48.751181 25626 catalog_manager.cc:4841] TS 8d9a4167a9f744f1b18c735265a626ac (127.9.77.131:42261): tablet ec35cd2b06534c46bc23516cfca3defe (table test-table [id=2acc34b9159e4a7e88517b482fdb8960]) successfully deleted
W20260810 01:47:48.761505 25622 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.77.129:35683 (error 108)
I20260810 01:47:48.762424 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 25959
I20260810 01:47:48.763067 26390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:47:48.762689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:35683 (local address 127.0.0.1:55540)
0810 01:47:48.762833 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 01:47:48.762920 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.129:35683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 01:47:48.772048 25622 catalog_manager.cc:4584] TS 67cf84cdf69a49e3bee56787823a2056 (127.9.77.129:35683): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ec35cd2b06534c46bc23516cfca3defe: Network error: Client connection negotiation failed: client connection to 127.9.77.129:35683: connect: Connection refused (error 111)
I20260810 01:47:48.779321 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 26166
I20260810 01:47:48.793334 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 25601
2026-08-10T01:47:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3059 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:47:48 dist-test-slave-kvfs krb5kdc[26431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:47:48 dist-test-slave-kvfs krb5kdc[26431](info): set up 2 sockets
Aug 10 01:47:48 dist-test-slave-kvfs krb5kdc[26431](info): commencing operation
krb5kdc: starting...
W20260810 01:47:50.150278 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.320s user 0.001s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:47:50 dist-test-slave-kvfs krb5kdc[26431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:47:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:47:50Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.77.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.77.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.77.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/rangeradmin_127.9.77.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.77.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/rangeradmin_127.9.77.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.77.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.77.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.77.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/rangerlookup_127.9.77.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.77.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/rangerlookup_127.9.77.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.77.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.77.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.77.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/HTTP_127.9.77.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.77.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/HTTP_127.9.77.148@KRBTEST.COM.keytab.
I20260810 01:47:50.372413 9526 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taski_V3Up/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/postgres -l logfile start
2026-08-10 01:47:53.179 UTC [26485] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 01:47:53.179 UTC [26485] LOG: listening on IPv4 address "127.9.77.148", port 46553
2026-08-10 01:47:53.183 UTC [26485] LOG: listening on Unix socket "/tmp/.s.PGSQL.46553"
2026-08-10 01:47:53.217 UTC [26494] LOG: database system was shut down at 2026-08-10 01:47:51 UTC
2026-08-10 01:47:53.222 UTC [26485] LOG: database system is ready to accept connections
I20260810 01:47:54.429204 9526 mini_postgres.cc:96] Postgres bound to 46553
2026-08-10 01:47:54.442 UTC [26508] FATAL: database "slave" does not exist
127.9.77.148:46553 - accepting connections
I20260810 01:47:54.443260 9526 mini_ranger.cc:161] Starting Ranger...
I20260810 01:47:54.464689 9526 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 01:47:55.032403 9526 mini_ranger.cc:87] Created ranger Postgres database
I20260810 01:47:55.032543 9526 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ranger-admin
2026-08-10 01:47:55,408 [I] DB FLAVOR :POSTGRES
2026-08-10 01:47:55,408 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 01:47:55,408 [I] Checking connection..
2026-08-10 01:47:55,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:46553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 01:47:55,688 [I] Checking connection passed.
2026-08-10 01:47:55,688 [I] --------- Verifying version history table ---------
2026-08-10 01:47:55,689 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:46553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 01:47:55,983 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 01:47:55,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:46553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 01:47:56,279 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 01:47:56,279 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 01:47:56,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:46553/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 01:47:56.560 UTC [26607] WARNING: there is no transaction in progress
2026-08-10 01:47:56,568 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 01:47:56,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:46553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 01:47:56,861 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 01:47:56,861 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 01:47:56,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:46553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 01:47:57,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:46553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 01:47:57,440 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:46553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-kvfs.c.gcp-upstream.internal','N') ;"
2026-08-10 01:47:57,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:46553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 01:47:58,008 [I] Table x_portal_user does not exist in database ranger
2026-08-10 01:47:58,009 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:46553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 01:47:58,320 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 01:47:58,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:46553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 01:47:58,586 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:46553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 01:47:58,865 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 01:47:58,865 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:46553/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 01:47:59.963 UTC [26787] WARNING: there is no transaction in progress
2026-08-10 01:47:59.981 UTC [26787] WARNING: there is no transaction in progress
2026-08-10 01:47:59.998 UTC [26787] WARNING: there is no transaction in progress
2026-08-10 01:48:00.016 UTC [26787] WARNING: there is no transaction in progress
2026-08-10 01:48:00.037 UTC [26787] WARNING: there is no transaction in progress
2026-08-10 01:48:00.055 UTC [26787] WARNING: there is no transaction in progress
2026-08-10 01:48:00.113 UTC [26787] WARNING: there is no transaction in progress
2026-08-10 01:48:00.121 UTC [26787] WARNING: there is no transaction in progress
2026-08-10 01:48:00.137 UTC [26787] WARNING: there is no transaction in progress
2026-08-10 01:48:00.148 UTC [26787] WARNING: there is no transaction in progress
2026-08-10 01:48:00.159 UTC [26787] WARNING: there is no transaction in progress
2026-08-10 01:48:00.173 UTC [26787] WARNING: there is no transaction in progress
2026-08-10 01:48:00.183 UTC [26787] WARNING: there is no transaction in progress
2026-08-10 01:48:00.191 UTC [26787] WARNING: there is no transaction in progress
2026-08-10 01:48:00.198 UTC [26787] WARNING: there is no transaction in progress
2026-08-10 01:48:00.205 UTC [26787] WARNING: there is no transaction in progress
2026-08-10 01:48:00,986 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 01:48:00,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:46553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 01:48:01,271 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 01:48:01,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:46553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 01:48:01,531 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 01:48:01,532 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:46553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 01:48:01,828 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 01:48:01,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:46553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 01:48:02,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:46553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 01:48:02,410 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:46553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-kvfs.c.gcp-upstream.internal';"
2026-08-10 01:48:02,703 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 01:48:02,703 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:46553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 01:48:02,995 [I] DB_PATCHES have already been applied
I20260810 01:48:03.002912 9526 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ranger-admin:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taski_V3Up/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 01:48:03.003011 9526 mini_ranger.cc:193] Using host: 127.9.77.148
I20260810 01:48:03.009195 9526 mini_ranger.cc:227] Ranger admin URL: http://127.9.77.148:58759
Aug 10, 2026 1:48:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 1:48:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 1:48:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 1:48:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 1:48:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 1:48:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 1:48:04 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 01:48:04.433097 9526 mini_ranger.cc:160] Time spent starting Ranger: real 9.990s user 0.003s sys 0.015s
Aug 10, 2026 1:48:27 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 1:48:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 1:48:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 1:48:27 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 1:48:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 1:48:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 1:48:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 1:48:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 1:48:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 1:48:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 1:48:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 1:48:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 1:48:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 1:48:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 1:48:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 1:48:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 1:48:27 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 1:48:28 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 01:48:29.153046 9526 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.77.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.77.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.77.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/rangerkms_127.9.77.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.77.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/rangerkms_127.9.77.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.77.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.77.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.77.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/HTTP_127.9.77.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.77.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/HTTP_127.9.77.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 01:48:29 dist-test-slave-kvfs krb5kdc[26431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326509, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 01:48:29.274668 9526 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 01:48:29.288894 9526 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 01:48:30.222693 9526 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 01:48:30.222847 9526 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ranger-kms
I20260810 01:48:30.223001 9526 mini_ranger_kms.cc:227] Using postgres at 127.9.77.148:46553
2026-08-10 01:48:31,067 [I] DB FLAVOR :POSTGRES
2026-08-10 01:48:31,067 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 01:48:31,067 [I] Checking connection
2026-08-10 01:48:31,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:46553/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 01:48:31,389 [I] connection success
2026-08-10 01:48:31,389 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 01:48:31,389 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 01:48:31,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:46553/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 01:48:31,695 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 01:48:31,695 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 01:48:31,695 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 01:48:31,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:46553/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 01:48:31,998 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 01:48:32.550884 9526 mini_ranger_kms.cc:310] Created kms service
I20260810 01:48:32.785323 9526 mini_ranger_kms.cc:326] Created kudu user
I20260810 01:48:32.848533 9526 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 01:48:32 dist-test-slave-kvfs krb5kdc[26431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326509, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.77.148@KRBTEST.COM
I20260810 01:48:33.342330 9526 mini_ranger_kms.cc:384] Added ranger policy
I20260810 01:48:33.342585 9526 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ranger-kms:/tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 01:48:33.342628 9526 mini_ranger_kms.cc:240] Using host: 127.9.77.148
I20260810 01:48:33.350313 9526 mini_ranger_kms.cc:276] Ranger KMS PID: 27053
I20260810 01:48:33.350520 9526 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.77.148:39597
Aug 10, 2026 1:48:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 1:48:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 1:48:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 1:48:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 1:48:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 1:48:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 01:48:35 dist-test-slave-kvfs krb5kdc[26431](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786326515, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.77.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:48:36.223676 9526 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.949s user 0.017s sys 0.405s
I20260810 01:48:36.223869 9526 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 01:48:36.224187 9526 mini_ranger_kms.cc:401] 127.9.77.148:39597/kms/v1/keys
Aug 10 01:48:36 dist-test-slave-kvfs krb5kdc[26431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326515, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.77.148@KRBTEST.COM for HTTP/127.9.77.148@KRBTEST.COM
Aug 10 01:48:37 dist-test-slave-kvfs krb5kdc[26431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326515, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.77.148@KRBTEST.COM for HTTP/127.9.77.148@KRBTEST.COM
Aug 10 01:48:39 dist-test-slave-kvfs krb5kdc[26431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:48:39.883668 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:41993
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:41895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:41993
--ranger_config_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.77.148:39597/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:48:40.036715 27133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:48:40.037086 27133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:48:40.037132 27133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:48:40.041568 27133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:48:40.041663 27133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:48:40.041687 27133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:48:40.041704 27133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:48:40.041720 27133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:48:40.048206 27133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:41895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.77.148:39597/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:41993
--ranger_config_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:41993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:48:40.049747 27133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:48:40.051147 27133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 01:48:40.062798 27133 server_base.cc:1034] running on GCE node
W20260810 01:48:40.062786 27139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:48:40.063030 27141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:48:40.062780 27145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:48:40.064191 27133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:48:40.065958 27133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:48:40.067174 27133 hybrid_clock.cc:648] HybridClock initialized: now 1786326520067143 us; error 45 us; skew 500 ppm
Aug 10 01:48:40 dist-test-slave-kvfs krb5kdc[26431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326520, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:48:40.071237 27133 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:48:40.072623 27133 webserver.cc:460] Webserver started at http://127.9.77.190:44979/ using document root <none> and password file <none>
I20260810 01:48:40.073348 27133 fs_manager.cc:359] Metadata directory not provided
I20260810 01:48:40.073410 27133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:48:40.073694 27133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 01:48:40 dist-test-slave-kvfs krb5kdc[26431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326520, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for HTTP/127.9.77.148@KRBTEST.COM
I20260810 01:48:40.145844 27133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "be827435f4ba4a0fbc9a735de5e8228e"
format_stamp: "Formatted at 2026-08-10 01:48:40 on dist-test-slave-kvfs"
server_key: "608c1ffa02d55649e893d5d3d3ee49c0"
server_key_iv: "5ad7e6b9d4fe442254e08e8269b32f44"
server_key_version: "kuduclusterkey@0"
I20260810 01:48:40.146685 27133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "be827435f4ba4a0fbc9a735de5e8228e"
format_stamp: "Formatted at 2026-08-10 01:48:40 on dist-test-slave-kvfs"
server_key: "608c1ffa02d55649e893d5d3d3ee49c0"
server_key_iv: "5ad7e6b9d4fe442254e08e8269b32f44"
server_key_version: "kuduclusterkey@0"
I20260810 01:48:40.151803 27133 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260810 01:48:40.170691 27157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:48:40.172621 27133 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 01:48:40.172798 27133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "be827435f4ba4a0fbc9a735de5e8228e"
format_stamp: "Formatted at 2026-08-10 01:48:40 on dist-test-slave-kvfs"
server_key: "608c1ffa02d55649e893d5d3d3ee49c0"
server_key_iv: "5ad7e6b9d4fe442254e08e8269b32f44"
server_key_version: "kuduclusterkey@0"
I20260810 01:48:40.172972 27133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:48:40.190156 27133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:48:40.193979 27133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:48:40.194242 27133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:48:40.206058 27133 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:41993
I20260810 01:48:40.206092 27241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:41993 every 8 connection(s)
I20260810 01:48:40.207391 27133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:48:40.211149 27242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:48:40.217432 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 27133
I20260810 01:48:40.217610 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:48:40.217962 9526 external_mini_cluster.cc:1428] Setting key 4aa635d028ff7c63c2b9fff9f9c463ea
I20260810 01:48:40.218822 27242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be827435f4ba4a0fbc9a735de5e8228e: Bootstrap starting.
I20260810 01:48:40.222177 27242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be827435f4ba4a0fbc9a735de5e8228e: Neither blocks nor log segments found. Creating new log.
I20260810 01:48:40.223304 27242 log.cc:824] T 00000000000000000000000000000000 P be827435f4ba4a0fbc9a735de5e8228e: Log is configured to *not* fsync() on all Append() calls
I20260810 01:48:40.226351 27242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be827435f4ba4a0fbc9a735de5e8228e: No bootstrap required, opened a new log
Aug 10 01:48:40 dist-test-slave-kvfs krb5kdc[26431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:48:40.232197 27242 raft_consensus.cc:348] T 00000000000000000000000000000000 P be827435f4ba4a0fbc9a735de5e8228e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be827435f4ba4a0fbc9a735de5e8228e" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 41993 } }
I20260810 01:48:40.232527 27242 raft_consensus.cc:374] T 00000000000000000000000000000000 P be827435f4ba4a0fbc9a735de5e8228e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:48:40.232606 27242 raft_consensus.cc:729] T 00000000000000000000000000000000 P be827435f4ba4a0fbc9a735de5e8228e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be827435f4ba4a0fbc9a735de5e8228e, State: Initialized, Role: FOLLOWER
I20260810 01:48:40.233203 27242 consensus_queue.cc:260] T 00000000000000000000000000000000 P be827435f4ba4a0fbc9a735de5e8228e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be827435f4ba4a0fbc9a735de5e8228e" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 41993 } }
I20260810 01:48:40.233388 27242 raft_consensus.cc:388] T 00000000000000000000000000000000 P be827435f4ba4a0fbc9a735de5e8228e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:48:40.233477 27242 raft_consensus.cc:482] T 00000000000000000000000000000000 P be827435f4ba4a0fbc9a735de5e8228e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:48:40.233631 27242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be827435f4ba4a0fbc9a735de5e8228e [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:48:40.234998 27242 raft_consensus.cc:504] T 00000000000000000000000000000000 P be827435f4ba4a0fbc9a735de5e8228e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be827435f4ba4a0fbc9a735de5e8228e" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 41993 } }
I20260810 01:48:40.235531 27242 leader_election.cc:302] T 00000000000000000000000000000000 P be827435f4ba4a0fbc9a735de5e8228e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be827435f4ba4a0fbc9a735de5e8228e; no voters:
I20260810 01:48:40.236081 27242 leader_election.cc:288] T 00000000000000000000000000000000 P be827435f4ba4a0fbc9a735de5e8228e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:48:40.236413 27248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be827435f4ba4a0fbc9a735de5e8228e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:48:40.236742 27248 raft_consensus.cc:686] T 00000000000000000000000000000000 P be827435f4ba4a0fbc9a735de5e8228e [term 1 LEADER]: Becoming Leader. State: Replica: be827435f4ba4a0fbc9a735de5e8228e, State: Running, Role: LEADER
I20260810 01:48:40.237236 27248 consensus_queue.cc:237] T 00000000000000000000000000000000 P be827435f4ba4a0fbc9a735de5e8228e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be827435f4ba4a0fbc9a735de5e8228e" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 41993 } }
I20260810 01:48:40.237928 27242 sys_catalog.cc:564] T 00000000000000000000000000000000 P be827435f4ba4a0fbc9a735de5e8228e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:48:40.239418 27249 sys_catalog.cc:455] T 00000000000000000000000000000000 P be827435f4ba4a0fbc9a735de5e8228e [sys.catalog]: SysCatalogTable state changed. Reason: New leader be827435f4ba4a0fbc9a735de5e8228e. Latest consensus state: current_term: 1 leader_uuid: "be827435f4ba4a0fbc9a735de5e8228e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be827435f4ba4a0fbc9a735de5e8228e" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 41993 } } }
I20260810 01:48:40.239749 27249 sys_catalog.cc:458] T 00000000000000000000000000000000 P be827435f4ba4a0fbc9a735de5e8228e [sys.catalog]: This master's current role is: LEADER
I20260810 01:48:40.240061 27245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:48:40.219970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39664 (local address 127.9.77.190:41993)
0810 01:48:40.220737 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0810 01:48:40.220762 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 01:48:40.220819 (+ 57us) server_negotiation.cc:407] Connection header received
0810 01:48:40.221946 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:48:40.221979 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:48:40.222594 (+ 615us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:48:40.223088 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:48:40.224334 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:48:40.225907 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:48:40.227405 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:48:40.228220 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:48:40.228450 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:48:40.232269 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:48:40.232277 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:48:40.232301 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:48:40.232351 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:48:40.234999 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:48:40.235653 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:48:40.235662 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:48:40.235671 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:48:40.235848 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:48:40.236281 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:48:40.236284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:48:40.236285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:48:40.236946 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:48:40.238055 (+ 1109us) server_negotiation.cc:1035] Waiting for connection context
0810 01:48:40.238241 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 01:48:40.238614 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":237,"threads_started":1}
I20260810 01:48:40.239075 27248 sys_catalog.cc:455] T 00000000000000000000000000000000 P be827435f4ba4a0fbc9a735de5e8228e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be827435f4ba4a0fbc9a735de5e8228e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be827435f4ba4a0fbc9a735de5e8228e" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 41993 } } }
I20260810 01:48:40.240625 27248 sys_catalog.cc:458] T 00000000000000000000000000000000 P be827435f4ba4a0fbc9a735de5e8228e [sys.catalog]: This master's current role is: LEADER
I20260810 01:48:40.241379 27242 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 01:48:40.243731 27270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:48:40.248121 27270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:48:40.267910 27270 catalog_manager.cc:1347] Generated new cluster ID: a8bc4f0996044af0aed27d434b1807df
I20260810 01:48:40.268047 27270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:48:40.302552 27270 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:48:40.306437 27270 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:48:40.316702 27270 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be827435f4ba4a0fbc9a735de5e8228e: Generated new TSK 0
I20260810 01:48:40.319775 27270 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.77.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
I20260810 01:48:41.171625 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.77.148:39597/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:0
--local_ip_for_outbound_sockets=127.9.77.129
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:41993
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:41895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:48:41.358399 27305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:48:41.358978 27305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:48:41.359117 27305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:48:41.363927 27305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:48:41.364069 27305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:48:41.364189 27305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:48:41.370368 27305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:41895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.77.148:39597/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:41993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:48:41.371930 27305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:48:41.373441 27305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:48:41.386629 27323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:48:41.386884 27326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:48:41.386525 27320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:48:41.387410 27305 server_base.cc:1034] running on GCE node
I20260810 01:48:41.387989 27305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:48:41.388932 27305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:48:41.390193 27305 hybrid_clock.cc:648] HybridClock initialized: now 1786326521390139 us; error 61 us; skew 500 ppm
Aug 10 01:48:41 dist-test-slave-kvfs krb5kdc[26431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326521, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:48:41.394049 27305 init.cc:377] Logged in from keytab as kudu/127.9.77.129@KRBTEST.COM (short username kudu)
I20260810 01:48:41.395397 27305 webserver.cc:460] Webserver started at http://127.9.77.129:36497/ using document root <none> and password file <none>
I20260810 01:48:41.396274 27305 fs_manager.cc:359] Metadata directory not provided
I20260810 01:48:41.396378 27305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:48:41.396677 27305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 01:48:41 dist-test-slave-kvfs krb5kdc[26431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326521, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for HTTP/127.9.77.148@KRBTEST.COM
I20260810 01:48:41.420522 27305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-0/data/instance:
uuid: "cc655dbacd6e45deae3b2d906bd63de5"
format_stamp: "Formatted at 2026-08-10 01:48:41 on dist-test-slave-kvfs"
server_key: "726cf8526ffbe025936c576e9ed8828c"
server_key_iv: "37fd8bf5d6b661e7859c40288192d5a6"
server_key_version: "kuduclusterkey@0"
I20260810 01:48:41.421588 27305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance:
uuid: "cc655dbacd6e45deae3b2d906bd63de5"
format_stamp: "Formatted at 2026-08-10 01:48:41 on dist-test-slave-kvfs"
server_key: "726cf8526ffbe025936c576e9ed8828c"
server_key_iv: "37fd8bf5d6b661e7859c40288192d5a6"
server_key_version: "kuduclusterkey@0"
I20260810 01:48:41.438958 27305 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.002s sys 0.004s
I20260810 01:48:41.468271 27338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:48:41.470849 27305 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260810 01:48:41.471068 27305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "cc655dbacd6e45deae3b2d906bd63de5"
format_stamp: "Formatted at 2026-08-10 01:48:41 on dist-test-slave-kvfs"
server_key: "726cf8526ffbe025936c576e9ed8828c"
server_key_iv: "37fd8bf5d6b661e7859c40288192d5a6"
server_key_version: "kuduclusterkey@0"
I20260810 01:48:41.471285 27305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:48:41.494439 27305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:48:41.499989 27305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:48:41.500403 27305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:48:41.501655 27305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:48:41.503034 27305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:48:41.503118 27305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:48:41.503229 27305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:48:41.503268 27305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:48:41.518548 27305 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:44587
I20260810 01:48:41.519459 27506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:44587 every 8 connection(s)
I20260810 01:48:41.520399 27305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:48:41.526006 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 27305
I20260810 01:48:41.526198 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance
I20260810 01:48:41.526571 9526 external_mini_cluster.cc:1428] Setting key 5846d27845d1ca0fb9467d44b4f2a8a6
Aug 10 01:48:41 dist-test-slave-kvfs krb5kdc[26431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326521, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.77.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.130@KRBTEST.COM" created.
I20260810 01:48:41.549999 27510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:48:41.523719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:54979 (local address 127.9.77.190:41993)
0810 01:48:41.524200 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0810 01:48:41.524207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:48:41.528093 (+ 3886us) server_negotiation.cc:407] Connection header received
0810 01:48:41.533538 (+ 5445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:48:41.533546 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:48:41.533638 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:48:41.533725 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:48:41.535757 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:48:41.536839 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:48:41.539189 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:48:41.539717 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:48:41.539825 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:48:41.544475 (+ 4650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:48:41.544481 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:48:41.544486 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:48:41.544535 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:48:41.546879 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:48:41.547589 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:48:41.547594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:48:41.547596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:48:41.547700 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:48:41.548293 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:48:41.548297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:48:41.548299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:48:41.548558 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:48:41.548756 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 01:48:41.549582 (+ 826us) server_negotiation.cc:299] Negotiation successful
0810 01:48:41.549758 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":176,"threads_started":1}
I20260810 01:48:41.553859 27511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:48:41.524451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:41993 (local address 127.9.77.129:54979)
0810 01:48:41.527845 (+ 3394us) negotiation.cc:107] Waiting for socket to connect
0810 01:48:41.527939 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0810 01:48:41.533109 (+ 5170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:48:41.534428 (+ 1319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:48:41.534439 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:48:41.534840 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:48:41.535554 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:48:41.535572 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:48:41.537057 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:48:41.537065 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:48:41.539004 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:48:41.539020 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:48:41.540209 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:48:41.540216 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:48:41.540542 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:48:41.541603 (+ 1061us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:48:41.541639 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:48:41.544144 (+ 2505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:48:41.547036 (+ 2892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:48:41.547052 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:48:41.547073 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:48:41.547424 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:48:41.547837 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:48:41.547840 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:48:41.547842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:48:41.548165 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:48:41.548690 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:48:41.548700 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:48:41.549234 (+ 534us) client_negotiation.cc:769] Sending connection context
0810 01:48:41.549518 (+ 284us) client_negotiation.cc:240] Negotiation successful
0810 01:48:41.549986 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2716,"thread_start_us":144,"threads_started":1}
I20260810 01:48:41.556900 27507 heartbeater.cc:347] Connected to a master server at 127.9.77.190:41993
I20260810 01:48:41.557278 27507 heartbeater.cc:464] Registering TS with master...
I20260810 01:48:41.558202 27507 heartbeater.cc:511] Master 127.9.77.190:41993 requested a full tablet report, sending...
I20260810 01:48:41.561161 27187 ts_manager.cc:194] Registered new tserver with Master: cc655dbacd6e45deae3b2d906bd63de5 (127.9.77.129:44587)
I20260810 01:48:41.563225 27187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.129@KRBTEST.COM'} at 127.9.77.129:54979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
I20260810 01:48:41.575846 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.77.148:39597/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:0
--local_ip_for_outbound_sockets=127.9.77.130
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:41993
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:41895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 01:48:41 dist-test-slave-kvfs krb5kdc[26431](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786326521, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 01:48:41.778999 27514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:48:41.779572 27514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:48:41.779652 27514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:48:41.786341 27514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:48:41.786464 27514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:48:41.786566 27514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:48:41.795715 27514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:41895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.77.148:39597/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:41993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:48:41.797677 27514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:48:41.799472 27514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:48:41.815939 27525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:48:41.816627 27520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:48:41.817173 27521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:48:41.817342 27514 server_base.cc:1034] running on GCE node
I20260810 01:48:41.818050 27514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:48:41.819118 27514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:48:41.820406 27514 hybrid_clock.cc:648] HybridClock initialized: now 1786326521820375 us; error 53 us; skew 500 ppm
Aug 10 01:48:41 dist-test-slave-kvfs krb5kdc[26431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326521, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:48:41.825691 27514 init.cc:377] Logged in from keytab as kudu/127.9.77.130@KRBTEST.COM (short username kudu)
I20260810 01:48:41.827301 27514 webserver.cc:460] Webserver started at http://127.9.77.130:45661/ using document root <none> and password file <none>
I20260810 01:48:41.828394 27514 fs_manager.cc:359] Metadata directory not provided
I20260810 01:48:41.828572 27514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:48:41.828972 27514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 01:48:41 dist-test-slave-kvfs krb5kdc[26431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326521, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for HTTP/127.9.77.148@KRBTEST.COM
I20260810 01:48:41.853416 27514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-1/data/instance:
uuid: "4f5d02efadb743fba86fc6a20bf3a429"
format_stamp: "Formatted at 2026-08-10 01:48:41 on dist-test-slave-kvfs"
server_key: "177d02e3bdc1eab74151e47d1cdccde9"
server_key_iv: "c8f13cb2675b1769707315f71da541c2"
server_key_version: "kuduclusterkey@0"
I20260810 01:48:41.854617 27514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance:
uuid: "4f5d02efadb743fba86fc6a20bf3a429"
format_stamp: "Formatted at 2026-08-10 01:48:41 on dist-test-slave-kvfs"
server_key: "177d02e3bdc1eab74151e47d1cdccde9"
server_key_iv: "c8f13cb2675b1769707315f71da541c2"
server_key_version: "kuduclusterkey@0"
I20260810 01:48:41.860574 27514 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 01:48:41.885756 27535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:48:41.888343 27514 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260810 01:48:41.888552 27514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "4f5d02efadb743fba86fc6a20bf3a429"
format_stamp: "Formatted at 2026-08-10 01:48:41 on dist-test-slave-kvfs"
server_key: "177d02e3bdc1eab74151e47d1cdccde9"
server_key_iv: "c8f13cb2675b1769707315f71da541c2"
server_key_version: "kuduclusterkey@0"
I20260810 01:48:41.888764 27514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:48:41.920960 27514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:48:41.926935 27514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:48:41.927277 27514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:48:41.928326 27514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:48:41.929759 27514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:48:41.929888 27514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260810 01:48:41.930012 27514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:48:41.930079 27514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:48:41.945662 27514 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:37805
I20260810 01:48:41.945683 27710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:37805 every 8 connection(s)
I20260810 01:48:41.947170 27514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:48:41.947645 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 27514
I20260810 01:48:41.947798 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance
I20260810 01:48:41.948248 9526 external_mini_cluster.cc:1428] Setting key 3d5728c997ebc09d6b7bce5736f6e7c3
WARNING: no policy specified for kudu/127.9.77.131@KRBTEST.COM; defaulting to no policy
Aug 10 01:48:41 dist-test-slave-kvfs krb5kdc[26431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326521, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.131@KRBTEST.COM" created.
I20260810 01:48:41.971830 27510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:48:41.950235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:55641 (local address 127.9.77.190:41993)
0810 01:48:41.950393 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 01:48:41.950399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:48:41.954617 (+ 4218us) server_negotiation.cc:407] Connection header received
0810 01:48:41.956316 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:48:41.956320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:48:41.956468 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:48:41.956566 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:48:41.958742 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:48:41.959781 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:48:41.962038 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:48:41.962495 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:48:41.962590 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:48:41.966534 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:48:41.966543 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:48:41.966546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:48:41.966610 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:48:41.968524 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:48:41.969338 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:48:41.969342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:48:41.969344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:48:41.969465 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:48:41.969965 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:48:41.969971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:48:41.969975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:48:41.970282 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:48:41.970391 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 01:48:41.971327 (+ 936us) server_negotiation.cc:299] Negotiation successful
0810 01:48:41.971541 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 01:48:41.974113 27715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:48:41.950365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:41993 (local address 127.9.77.130:55641)
0810 01:48:41.954394 (+ 4029us) negotiation.cc:107] Waiting for socket to connect
0810 01:48:41.954481 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0810 01:48:41.955983 (+ 1502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:48:41.957077 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:48:41.957093 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:48:41.957592 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:48:41.958517 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:48:41.958544 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:48:41.959998 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:48:41.960005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:48:41.961853 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:48:41.961869 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:48:41.962885 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:48:41.962891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:48:41.963190 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:48:41.964120 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:48:41.964155 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:48:41.966256 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:48:41.968690 (+ 2434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:48:41.968701 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:48:41.968728 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:48:41.969187 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:48:41.969599 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:48:41.969604 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:48:41.969609 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:48:41.969849 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:48:41.970472 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:48:41.970483 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:48:41.970984 (+ 501us) client_negotiation.cc:769] Sending connection context
0810 01:48:41.972094 (+ 1110us) client_negotiation.cc:240] Negotiation successful
0810 01:48:41.972613 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3689,"thread_start_us":203,"threads_started":1}
I20260810 01:48:41.975895 27711 heartbeater.cc:347] Connected to a master server at 127.9.77.190:41993
I20260810 01:48:41.976244 27711 heartbeater.cc:464] Registering TS with master...
I20260810 01:48:41.977025 27711 heartbeater.cc:511] Master 127.9.77.190:41993 requested a full tablet report, sending...
I20260810 01:48:41.978722 27187 ts_manager.cc:194] Registered new tserver with Master: 4f5d02efadb743fba86fc6a20bf3a429 (127.9.77.130:37805)
I20260810 01:48:41.979591 27187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.130@KRBTEST.COM'} at 127.9.77.130:55641
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
I20260810 01:48:41.996570 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.77.148:39597/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:0
--local_ip_for_outbound_sockets=127.9.77.131
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:41993
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:41895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 01:48:42 dist-test-slave-kvfs krb5kdc[26431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326521, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for HTTP/127.9.77.148@KRBTEST.COM
W20260810 01:48:42.218765 27718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:48:42.219226 27718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:48:42.219296 27718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:48:42.225522 27718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:48:42.225697 27718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:48:42.225836 27718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:48:42.234720 27718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:41895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.77.148:39597/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:41993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:48:42.236773 27718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:48:42.238476 27718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:48:42.254477 27723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:48:42.254938 27727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:48:42.255136 27718 server_base.cc:1034] running on GCE node
W20260810 01:48:42.255221 27724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:48:42.256202 27718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:48:42.257368 27718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:48:42.258740 27718 hybrid_clock.cc:648] HybridClock initialized: now 1786326522258712 us; error 55 us; skew 500 ppm
Aug 10 01:48:42 dist-test-slave-kvfs krb5kdc[26431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326522, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:48:42.263360 27718 init.cc:377] Logged in from keytab as kudu/127.9.77.131@KRBTEST.COM (short username kudu)
I20260810 01:48:42.264889 27718 webserver.cc:460] Webserver started at http://127.9.77.131:35501/ using document root <none> and password file <none>
I20260810 01:48:42.265892 27718 fs_manager.cc:359] Metadata directory not provided
I20260810 01:48:42.265995 27718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:48:42.266355 27718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 01:48:42 dist-test-slave-kvfs krb5kdc[26431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326522, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for HTTP/127.9.77.148@KRBTEST.COM
I20260810 01:48:42.297188 27718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-2/data/instance:
uuid: "08bae345660b4b8286adabf142e8b00e"
format_stamp: "Formatted at 2026-08-10 01:48:42 on dist-test-slave-kvfs"
server_key: "c51324b9fe2a1e91cfbe1a2d1dc51c16"
server_key_iv: "99016b7f90da87709aac39fa79a06e77"
server_key_version: "kuduclusterkey@0"
I20260810 01:48:42.298235 27718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance:
uuid: "08bae345660b4b8286adabf142e8b00e"
format_stamp: "Formatted at 2026-08-10 01:48:42 on dist-test-slave-kvfs"
server_key: "c51324b9fe2a1e91cfbe1a2d1dc51c16"
server_key_iv: "99016b7f90da87709aac39fa79a06e77"
server_key_version: "kuduclusterkey@0"
I20260810 01:48:42.305519 27718 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260810 01:48:42.336674 27739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:48:42.339685 27718 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.005s sys 0.001s
I20260810 01:48:42.339905 27718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "08bae345660b4b8286adabf142e8b00e"
format_stamp: "Formatted at 2026-08-10 01:48:42 on dist-test-slave-kvfs"
server_key: "c51324b9fe2a1e91cfbe1a2d1dc51c16"
server_key_iv: "99016b7f90da87709aac39fa79a06e77"
server_key_version: "kuduclusterkey@0"
I20260810 01:48:42.340171 27718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:48:42.364138 27718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:48:42.369195 27718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:48:42.369499 27718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:48:42.370643 27718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:48:42.372256 27718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:48:42.372346 27718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:48:42.372430 27718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:48:42.372470 27718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:48:42.393929 27718 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:40927
I20260810 01:48:42.394505 27934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:40927 every 8 connection(s)
I20260810 01:48:42.395568 27718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:48:42.400470 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 27718
I20260810 01:48:42.400645 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance
I20260810 01:48:42.401031 9526 external_mini_cluster.cc:1428] Setting key ef390e93d40034bbe594300737ef363c
Aug 10 01:48:42 dist-test-slave-kvfs krb5kdc[26431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326522, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:48:42.417563 27510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:48:42.401546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:37535 (local address 127.9.77.190:41993)
0810 01:48:42.402066 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0810 01:48:42.402073 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:48:42.402092 (+ 19us) server_negotiation.cc:407] Connection header received
0810 01:48:42.402152 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:48:42.402155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:48:42.402214 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:48:42.402313 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:48:42.404360 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:48:42.405215 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:48:42.407205 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:48:42.407591 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:48:42.407682 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:48:42.411876 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:48:42.411881 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:48:42.411884 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:48:42.411921 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:48:42.414030 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:48:42.414823 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:48:42.414827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:48:42.414829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:48:42.414937 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:48:42.415457 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:48:42.415461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:48:42.415462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:48:42.415751 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:48:42.415847 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 01:48:42.417178 (+ 1331us) server_negotiation.cc:299] Negotiation successful
0810 01:48:42.417365 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394}
I20260810 01:48:42.420145 27939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:48:42.399313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:41993 (local address 127.9.77.131:37535)
0810 01:48:42.400144 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0810 01:48:42.400225 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0810 01:48:42.401542 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:48:42.402636 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:48:42.402647 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:48:42.403150 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:48:42.404142 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:48:42.404169 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:48:42.405424 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:48:42.405427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:48:42.407038 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:48:42.407052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:48:42.407945 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:48:42.407949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:48:42.408247 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:48:42.409188 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:48:42.409218 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:48:42.411602 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:48:42.414198 (+ 2596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:48:42.414206 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:48:42.414233 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:48:42.414688 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:48:42.415067 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:48:42.415072 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:48:42.415075 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:48:42.415328 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:48:42.415929 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:48:42.415939 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:48:42.416813 (+ 874us) client_negotiation.cc:769] Sending connection context
0810 01:48:42.418082 (+ 1269us) client_negotiation.cc:240] Negotiation successful
0810 01:48:42.418577 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":206,"threads_started":1}
I20260810 01:48:42.421991 27936 heartbeater.cc:347] Connected to a master server at 127.9.77.190:41993
I20260810 01:48:42.422374 27936 heartbeater.cc:464] Registering TS with master...
I20260810 01:48:42.423218 27936 heartbeater.cc:511] Master 127.9.77.190:41993 requested a full tablet report, sending...
I20260810 01:48:42.424865 27187 ts_manager.cc:194] Registered new tserver with Master: 08bae345660b4b8286adabf142e8b00e (127.9.77.131:40927)
I20260810 01:48:42.425761 27187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.131@KRBTEST.COM'} at 127.9.77.131:37535
I20260810 01:48:42.428725 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 01:48:42.443104 27510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:48:42.431600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60274 (local address 127.9.77.190:41993)
0810 01:48:42.431743 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 01:48:42.431748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:48:42.432115 (+ 367us) server_negotiation.cc:407] Connection header received
0810 01:48:42.432209 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:48:42.432213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:48:42.432281 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:48:42.432391 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:48:42.433461 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:48:42.434347 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:48:42.435602 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:48:42.435968 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:48:42.436135 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:48:42.438432 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:48:42.438437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:48:42.438440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:48:42.438478 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:48:42.440781 (+ 2303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:48:42.441404 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:48:42.441408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:48:42.441410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:48:42.441523 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:48:42.441855 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:48:42.441859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:48:42.441861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:48:42.442148 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:48:42.442267 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 01:48:42.442713 (+ 446us) server_negotiation.cc:299] Negotiation successful
0810 01:48:42.442887 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 01:48:42.448920 27187 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 01:48:42.454753 27187 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:48:42.481546 27961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:48:42.468904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:40927 (local address 127.0.0.1:34048)
0810 01:48:42.471662 (+ 2758us) negotiation.cc:107] Waiting for socket to connect
0810 01:48:42.471747 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0810 01:48:42.471918 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:48:42.475441 (+ 3523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:48:42.475445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:48:42.475465 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:48:42.475591 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:48:42.475598 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:48:42.477395 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:48:42.477402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:48:42.479133 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:48:42.479146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:48:42.480837 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:48:42.480842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:48:42.480978 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:48:42.481030 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 01:48:42.481170 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 01:48:42.481338 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2516,"mutex_wait_us":139,"thread_start_us":159,"threads_started":1}
I20260810 01:48:42.482342 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:48:42.470033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34048 (local address 127.9.77.131:40927)
0810 01:48:42.474890 (+ 4857us) server_negotiation.cc:206] Beginning negotiation
0810 01:48:42.474906 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 01:48:42.474937 (+ 31us) server_negotiation.cc:407] Connection header received
0810 01:48:42.475032 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:48:42.475038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:48:42.475241 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:48:42.475775 (+ 534us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:48:42.475961 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:48:42.477246 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:48:42.479323 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:48:42.480432 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:48:42.480555 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:48:42.480765 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 01:48:42.482026 (+ 1261us) server_negotiation.cc:299] Negotiation successful
0810 01:48:42.482168 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4712,"thread_start_us":171,"threads_started":1}
I20260810 01:48:42.484522 27956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:48:42.468988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:44587 (local address 127.0.0.1:57716)
0810 01:48:42.472234 (+ 3246us) negotiation.cc:107] Waiting for socket to connect
0810 01:48:42.472248 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 01:48:42.472337 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:48:42.473178 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:48:42.473959 (+ 781us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:48:42.473994 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:48:42.474199 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:48:42.474208 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:48:42.477473 (+ 3265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:48:42.477476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:48:42.480280 (+ 2804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:48:42.480293 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:48:42.483967 (+ 3674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:48:42.483971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:48:42.484145 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:48:42.484173 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 01:48:42.484260 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 01:48:42.484351 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3177,"mutex_wait_us":176,"thread_start_us":143,"threads_started":1}
I20260810 01:48:42.485196 27957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:48:42.469233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57716 (local address 127.9.77.129:44587)
0810 01:48:42.472329 (+ 3096us) server_negotiation.cc:206] Beginning negotiation
0810 01:48:42.472337 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 01:48:42.472371 (+ 34us) server_negotiation.cc:407] Connection header received
0810 01:48:42.472479 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:48:42.472486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:48:42.472766 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:48:42.472950 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:48:42.474374 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:48:42.477337 (+ 2963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:48:42.482575 (+ 5238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:48:42.483574 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:48:42.483700 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:48:42.483881 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 01:48:42.484836 (+ 955us) server_negotiation.cc:299] Negotiation successful
0810 01:48:42.484984 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2956,"thread_start_us":104,"threads_started":1}
I20260810 01:48:42.487726 27810 tablet_service.cc:1467] Processing CreateTablet for tablet a6f3fca3751b4b2986ad787adc4a50e1 (DEFAULT_TABLE table=test-table [id=e0a3799b32d9411baca322790f76949d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:48:42.489143 27953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:48:42.470257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:37805 (local address 127.0.0.1:48486)
0810 01:48:42.474453 (+ 4196us) negotiation.cc:107] Waiting for socket to connect
0810 01:48:42.474467 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 01:48:42.474567 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:48:42.475583 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:48:42.475587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:48:42.475605 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:48:42.475696 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:48:42.475701 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:48:42.478398 (+ 2697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:48:42.478402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:48:42.481515 (+ 3113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:48:42.481526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:48:42.488489 (+ 6963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:48:42.488494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:48:42.488731 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:48:42.488764 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 01:48:42.488849 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 01:48:42.488950 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4123,"mutex_wait_us":3,"thread_start_us":118,"threads_started":1}
I20260810 01:48:42.489424 27810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6f3fca3751b4b2986ad787adc4a50e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:48:42.489020 27386 tablet_service.cc:1467] Processing CreateTablet for tablet a6f3fca3751b4b2986ad787adc4a50e1 (DEFAULT_TABLE table=test-table [id=e0a3799b32d9411baca322790f76949d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:48:42.490075 27962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:48:42.470766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48486 (local address 127.9.77.130:37805)
0810 01:48:42.471466 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0810 01:48:42.471490 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 01:48:42.474575 (+ 3085us) server_negotiation.cc:407] Connection header received
0810 01:48:42.474709 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:48:42.474716 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:48:42.475000 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:48:42.475398 (+ 398us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:48:42.475828 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:48:42.478176 (+ 2348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:48:42.482010 (+ 3834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:48:42.487993 (+ 5983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:48:42.488166 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:48:42.488387 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 01:48:42.489764 (+ 1377us) server_negotiation.cc:299] Negotiation successful
0810 01:48:42.489908 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":130,"threads_started":1}
I20260810 01:48:42.490211 27386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6f3fca3751b4b2986ad787adc4a50e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:48:42.495237 27577 tablet_service.cc:1467] Processing CreateTablet for tablet a6f3fca3751b4b2986ad787adc4a50e1 (DEFAULT_TABLE table=test-table [id=e0a3799b32d9411baca322790f76949d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:48:42.496757 27577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6f3fca3751b4b2986ad787adc4a50e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:48:42.510385 27966 tablet_bootstrap.cc:492] T a6f3fca3751b4b2986ad787adc4a50e1 P cc655dbacd6e45deae3b2d906bd63de5: Bootstrap starting.
I20260810 01:48:42.511499 27968 tablet_bootstrap.cc:492] T a6f3fca3751b4b2986ad787adc4a50e1 P 4f5d02efadb743fba86fc6a20bf3a429: Bootstrap starting.
I20260810 01:48:42.514832 27968 tablet_bootstrap.cc:654] T a6f3fca3751b4b2986ad787adc4a50e1 P 4f5d02efadb743fba86fc6a20bf3a429: Neither blocks nor log segments found. Creating new log.
I20260810 01:48:42.514832 27966 tablet_bootstrap.cc:654] T a6f3fca3751b4b2986ad787adc4a50e1 P cc655dbacd6e45deae3b2d906bd63de5: Neither blocks nor log segments found. Creating new log.
I20260810 01:48:42.516114 27968 log.cc:824] T a6f3fca3751b4b2986ad787adc4a50e1 P 4f5d02efadb743fba86fc6a20bf3a429: Log is configured to *not* fsync() on all Append() calls
I20260810 01:48:42.517896 27965 tablet_bootstrap.cc:492] T a6f3fca3751b4b2986ad787adc4a50e1 P 08bae345660b4b8286adabf142e8b00e: Bootstrap starting.
Aug 10 01:48:42 dist-test-slave-kvfs krb5kdc[26431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326521, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for HTTP/127.9.77.148@KRBTEST.COM
I20260810 01:48:42.519228 27968 tablet_bootstrap.cc:492] T a6f3fca3751b4b2986ad787adc4a50e1 P 4f5d02efadb743fba86fc6a20bf3a429: No bootstrap required, opened a new log
I20260810 01:48:42.519503 27968 ts_tablet_manager.cc:1397] T a6f3fca3751b4b2986ad787adc4a50e1 P 4f5d02efadb743fba86fc6a20bf3a429: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260810 01:48:42.523242 27966 log.cc:824] T a6f3fca3751b4b2986ad787adc4a50e1 P cc655dbacd6e45deae3b2d906bd63de5: Log is configured to *not* fsync() on all Append() calls
I20260810 01:48:42.525368 27968 raft_consensus.cc:348] T a6f3fca3751b4b2986ad787adc4a50e1 P 4f5d02efadb743fba86fc6a20bf3a429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bae345660b4b8286adabf142e8b00e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 40927 } } peers { permanent_uuid: "cc655dbacd6e45deae3b2d906bd63de5" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 44587 } } peers { permanent_uuid: "4f5d02efadb743fba86fc6a20bf3a429" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37805 } }
I20260810 01:48:42.525735 27968 raft_consensus.cc:374] T a6f3fca3751b4b2986ad787adc4a50e1 P 4f5d02efadb743fba86fc6a20bf3a429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:48:42.525790 27968 raft_consensus.cc:729] T a6f3fca3751b4b2986ad787adc4a50e1 P 4f5d02efadb743fba86fc6a20bf3a429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5d02efadb743fba86fc6a20bf3a429, State: Initialized, Role: FOLLOWER
I20260810 01:48:42.526458 27968 consensus_queue.cc:260] T a6f3fca3751b4b2986ad787adc4a50e1 P 4f5d02efadb743fba86fc6a20bf3a429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bae345660b4b8286adabf142e8b00e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 40927 } } peers { permanent_uuid: "cc655dbacd6e45deae3b2d906bd63de5" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 44587 } } peers { permanent_uuid: "4f5d02efadb743fba86fc6a20bf3a429" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37805 } }
I20260810 01:48:42.527863 27968 ts_tablet_manager.cc:1428] T a6f3fca3751b4b2986ad787adc4a50e1 P 4f5d02efadb743fba86fc6a20bf3a429: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260810 01:48:42.528369 27711 heartbeater.cc:503] Master 127.9.77.190:41993 was elected leader, sending a full tablet report...
I20260810 01:48:42.538297 27966 tablet_bootstrap.cc:492] T a6f3fca3751b4b2986ad787adc4a50e1 P cc655dbacd6e45deae3b2d906bd63de5: No bootstrap required, opened a new log
I20260810 01:48:42.538641 27966 ts_tablet_manager.cc:1397] T a6f3fca3751b4b2986ad787adc4a50e1 P cc655dbacd6e45deae3b2d906bd63de5: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.000s
I20260810 01:48:42.544710 27965 tablet_bootstrap.cc:654] T a6f3fca3751b4b2986ad787adc4a50e1 P 08bae345660b4b8286adabf142e8b00e: Neither blocks nor log segments found. Creating new log.
I20260810 01:48:42.544548 27966 raft_consensus.cc:348] T a6f3fca3751b4b2986ad787adc4a50e1 P cc655dbacd6e45deae3b2d906bd63de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bae345660b4b8286adabf142e8b00e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 40927 } } peers { permanent_uuid: "cc655dbacd6e45deae3b2d906bd63de5" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 44587 } } peers { permanent_uuid: "4f5d02efadb743fba86fc6a20bf3a429" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37805 } }
I20260810 01:48:42.544994 27966 raft_consensus.cc:374] T a6f3fca3751b4b2986ad787adc4a50e1 P cc655dbacd6e45deae3b2d906bd63de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:48:42.545073 27966 raft_consensus.cc:729] T a6f3fca3751b4b2986ad787adc4a50e1 P cc655dbacd6e45deae3b2d906bd63de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc655dbacd6e45deae3b2d906bd63de5, State: Initialized, Role: FOLLOWER
I20260810 01:48:42.545768 27966 consensus_queue.cc:260] T a6f3fca3751b4b2986ad787adc4a50e1 P cc655dbacd6e45deae3b2d906bd63de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bae345660b4b8286adabf142e8b00e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 40927 } } peers { permanent_uuid: "cc655dbacd6e45deae3b2d906bd63de5" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 44587 } } peers { permanent_uuid: "4f5d02efadb743fba86fc6a20bf3a429" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37805 } }
I20260810 01:48:42.546363 27965 log.cc:824] T a6f3fca3751b4b2986ad787adc4a50e1 P 08bae345660b4b8286adabf142e8b00e: Log is configured to *not* fsync() on all Append() calls
I20260810 01:48:42.547926 27966 ts_tablet_manager.cc:1428] T a6f3fca3751b4b2986ad787adc4a50e1 P cc655dbacd6e45deae3b2d906bd63de5: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260810 01:48:42.548514 27507 heartbeater.cc:503] Master 127.9.77.190:41993 was elected leader, sending a full tablet report...
I20260810 01:48:42.556743 27965 tablet_bootstrap.cc:492] T a6f3fca3751b4b2986ad787adc4a50e1 P 08bae345660b4b8286adabf142e8b00e: No bootstrap required, opened a new log
I20260810 01:48:42.557097 27965 ts_tablet_manager.cc:1397] T a6f3fca3751b4b2986ad787adc4a50e1 P 08bae345660b4b8286adabf142e8b00e: Time spent bootstrapping tablet: real 0.045s user 0.010s sys 0.000s
I20260810 01:48:42.563738 27965 raft_consensus.cc:348] T a6f3fca3751b4b2986ad787adc4a50e1 P 08bae345660b4b8286adabf142e8b00e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bae345660b4b8286adabf142e8b00e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 40927 } } peers { permanent_uuid: "cc655dbacd6e45deae3b2d906bd63de5" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 44587 } } peers { permanent_uuid: "4f5d02efadb743fba86fc6a20bf3a429" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37805 } }
I20260810 01:48:42.564337 27965 raft_consensus.cc:374] T a6f3fca3751b4b2986ad787adc4a50e1 P 08bae345660b4b8286adabf142e8b00e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:48:42.564438 27965 raft_consensus.cc:729] T a6f3fca3751b4b2986ad787adc4a50e1 P 08bae345660b4b8286adabf142e8b00e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bae345660b4b8286adabf142e8b00e, State: Initialized, Role: FOLLOWER
I20260810 01:48:42.565148 27965 consensus_queue.cc:260] T a6f3fca3751b4b2986ad787adc4a50e1 P 08bae345660b4b8286adabf142e8b00e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bae345660b4b8286adabf142e8b00e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 40927 } } peers { permanent_uuid: "cc655dbacd6e45deae3b2d906bd63de5" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 44587 } } peers { permanent_uuid: "4f5d02efadb743fba86fc6a20bf3a429" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37805 } }
I20260810 01:48:42.566869 27965 ts_tablet_manager.cc:1428] T a6f3fca3751b4b2986ad787adc4a50e1 P 08bae345660b4b8286adabf142e8b00e: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260810 01:48:42.567440 27936 heartbeater.cc:503] Master 127.9.77.190:41993 was elected leader, sending a full tablet report...
W20260810 01:48:42.656306 27937 tablet.cc:2369] T a6f3fca3751b4b2986ad787adc4a50e1 P 08bae345660b4b8286adabf142e8b00e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 01:48:42.703629 27714 tablet.cc:2369] T a6f3fca3751b4b2986ad787adc4a50e1 P 4f5d02efadb743fba86fc6a20bf3a429: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 01:48:42.779485 27508 tablet.cc:2369] T a6f3fca3751b4b2986ad787adc4a50e1 P cc655dbacd6e45deae3b2d906bd63de5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 01:48:42.788691 27978 raft_consensus.cc:482] T a6f3fca3751b4b2986ad787adc4a50e1 P cc655dbacd6e45deae3b2d906bd63de5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:48:42.789150 27978 raft_consensus.cc:504] T a6f3fca3751b4b2986ad787adc4a50e1 P cc655dbacd6e45deae3b2d906bd63de5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bae345660b4b8286adabf142e8b00e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 40927 } } peers { permanent_uuid: "cc655dbacd6e45deae3b2d906bd63de5" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 44587 } } peers { permanent_uuid: "4f5d02efadb743fba86fc6a20bf3a429" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37805 } }
I20260810 01:48:42.791074 27978 leader_election.cc:288] T a6f3fca3751b4b2986ad787adc4a50e1 P cc655dbacd6e45deae3b2d906bd63de5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08bae345660b4b8286adabf142e8b00e (127.9.77.131:40927), 4f5d02efadb743fba86fc6a20bf3a429 (127.9.77.130:37805)
I20260810 01:48:42.799006 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:48:42.792881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:36517 (local address 127.9.77.131:40927)
0810 01:48:42.793049 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 01:48:42.793054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:48:42.793661 (+ 607us) server_negotiation.cc:407] Connection header received
0810 01:48:42.793962 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:48:42.793967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:48:42.794049 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:48:42.794111 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:48:42.794944 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:48:42.795761 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:48:42.797576 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:48:42.798181 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:48:42.798289 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:48:42.798358 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 01:48:42.798730 (+ 372us) server_negotiation.cc:299] Negotiation successful
0810 01:48:42.798832 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 01:48:42.799566 27991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:48:42.791704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:40927 (local address 127.9.77.129:36517)
0810 01:48:42.793512 (+ 1808us) negotiation.cc:107] Waiting for socket to connect
0810 01:48:42.793530 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 01:48:42.793776 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:48:42.794322 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:48:42.794325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:48:42.794350 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:48:42.794773 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:48:42.794785 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:48:42.796088 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:48:42.796092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:48:42.797431 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:48:42.797443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:48:42.798476 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:48:42.798480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:48:42.798579 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:48:42.798613 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 01:48:42.799285 (+ 672us) client_negotiation.cc:240] Negotiation successful
0810 01:48:42.799373 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1724,"thread_start_us":230,"threads_started":1}
I20260810 01:48:42.800508 27840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6f3fca3751b4b2986ad787adc4a50e1" candidate_uuid: "cc655dbacd6e45deae3b2d906bd63de5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bae345660b4b8286adabf142e8b00e" is_pre_election: true
I20260810 01:48:42.801003 27840 raft_consensus.cc:2454] T a6f3fca3751b4b2986ad787adc4a50e1 P 08bae345660b4b8286adabf142e8b00e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc655dbacd6e45deae3b2d906bd63de5 in term 0.
I20260810 01:48:42.801688 27344 leader_election.cc:302] T a6f3fca3751b4b2986ad787adc4a50e1 P cc655dbacd6e45deae3b2d906bd63de5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bae345660b4b8286adabf142e8b00e, cc655dbacd6e45deae3b2d906bd63de5; no voters:
I20260810 01:48:42.802152 27978 raft_consensus.cc:2781] T a6f3fca3751b4b2986ad787adc4a50e1 P cc655dbacd6e45deae3b2d906bd63de5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:48:42.802268 27978 raft_consensus.cc:482] T a6f3fca3751b4b2986ad787adc4a50e1 P cc655dbacd6e45deae3b2d906bd63de5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:48:42.802322 27978 raft_consensus.cc:3037] T a6f3fca3751b4b2986ad787adc4a50e1 P cc655dbacd6e45deae3b2d906bd63de5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:48:42.804767 27978 raft_consensus.cc:504] T a6f3fca3751b4b2986ad787adc4a50e1 P cc655dbacd6e45deae3b2d906bd63de5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bae345660b4b8286adabf142e8b00e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 40927 } } peers { permanent_uuid: "cc655dbacd6e45deae3b2d906bd63de5" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 44587 } } peers { permanent_uuid: "4f5d02efadb743fba86fc6a20bf3a429" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37805 } }
I20260810 01:48:42.805292 27978 leader_election.cc:288] T a6f3fca3751b4b2986ad787adc4a50e1 P cc655dbacd6e45deae3b2d906bd63de5 [CANDIDATE]: Term 1 election: Requested vote from peers 08bae345660b4b8286adabf142e8b00e (127.9.77.131:40927), 4f5d02efadb743fba86fc6a20bf3a429 (127.9.77.130:37805)
I20260810 01:48:42.805925 27840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6f3fca3751b4b2986ad787adc4a50e1" candidate_uuid: "cc655dbacd6e45deae3b2d906bd63de5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08bae345660b4b8286adabf142e8b00e"
I20260810 01:48:42.806106 27840 raft_consensus.cc:3037] T a6f3fca3751b4b2986ad787adc4a50e1 P 08bae345660b4b8286adabf142e8b00e [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:48:42.806843 27962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:48:42.793086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:49705 (local address 127.9.77.130:37805)
0810 01:48:42.793273 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 01:48:42.793277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:48:42.797002 (+ 3725us) server_negotiation.cc:407] Connection header received
0810 01:48:42.799619 (+ 2617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:48:42.799623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:48:42.799710 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:48:42.799770 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:48:42.800458 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:48:42.801181 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:48:42.805059 (+ 3878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:48:42.805776 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:48:42.805885 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:48:42.805955 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 01:48:42.806556 (+ 601us) server_negotiation.cc:299] Negotiation successful
0810 01:48:42.806668 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260810 01:48:42.807528 27992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:48:42.792306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:37805 (local address 127.9.77.129:49705)
0810 01:48:42.796888 (+ 4582us) negotiation.cc:107] Waiting for socket to connect
0810 01:48:42.796905 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 01:48:42.799442 (+ 2537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:48:42.800163 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:48:42.800166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:48:42.800188 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:48:42.800307 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:48:42.800324 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:48:42.801478 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:48:42.801482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:48:42.803158 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:48:42.803180 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:48:42.806160 (+ 2980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:48:42.806163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:48:42.806373 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:48:42.806413 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 01:48:42.807205 (+ 792us) client_negotiation.cc:240] Negotiation successful
0810 01:48:42.807338 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4505,"thread_start_us":176,"threads_started":1}
I20260810 01:48:42.808406 27840 raft_consensus.cc:2454] T a6f3fca3751b4b2986ad787adc4a50e1 P 08bae345660b4b8286adabf142e8b00e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc655dbacd6e45deae3b2d906bd63de5 in term 1.
I20260810 01:48:42.808930 27344 leader_election.cc:302] T a6f3fca3751b4b2986ad787adc4a50e1 P cc655dbacd6e45deae3b2d906bd63de5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bae345660b4b8286adabf142e8b00e, cc655dbacd6e45deae3b2d906bd63de5; no voters:
I20260810 01:48:42.810009 27640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6f3fca3751b4b2986ad787adc4a50e1" candidate_uuid: "cc655dbacd6e45deae3b2d906bd63de5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5d02efadb743fba86fc6a20bf3a429" is_pre_election: true
I20260810 01:48:42.810441 27640 raft_consensus.cc:2454] T a6f3fca3751b4b2986ad787adc4a50e1 P 4f5d02efadb743fba86fc6a20bf3a429 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc655dbacd6e45deae3b2d906bd63de5 in term 0.
I20260810 01:48:42.811000 27638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6f3fca3751b4b2986ad787adc4a50e1" candidate_uuid: "cc655dbacd6e45deae3b2d906bd63de5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5d02efadb743fba86fc6a20bf3a429"
I20260810 01:48:42.811146 27638 raft_consensus.cc:3037] T a6f3fca3751b4b2986ad787adc4a50e1 P 4f5d02efadb743fba86fc6a20bf3a429 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:48:42.813706 27638 raft_consensus.cc:2454] T a6f3fca3751b4b2986ad787adc4a50e1 P 4f5d02efadb743fba86fc6a20bf3a429 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc655dbacd6e45deae3b2d906bd63de5 in term 1.
I20260810 01:48:42.814790 27978 raft_consensus.cc:2781] T a6f3fca3751b4b2986ad787adc4a50e1 P cc655dbacd6e45deae3b2d906bd63de5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:48:42.815240 27978 raft_consensus.cc:686] T a6f3fca3751b4b2986ad787adc4a50e1 P cc655dbacd6e45deae3b2d906bd63de5 [term 1 LEADER]: Becoming Leader. State: Replica: cc655dbacd6e45deae3b2d906bd63de5, State: Running, Role: LEADER
I20260810 01:48:42.815775 27978 consensus_queue.cc:237] T a6f3fca3751b4b2986ad787adc4a50e1 P cc655dbacd6e45deae3b2d906bd63de5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bae345660b4b8286adabf142e8b00e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 40927 } } peers { permanent_uuid: "cc655dbacd6e45deae3b2d906bd63de5" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 44587 } } peers { permanent_uuid: "4f5d02efadb743fba86fc6a20bf3a429" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37805 } }
I20260810 01:48:42.821100 27185 catalog_manager.cc:5495] T a6f3fca3751b4b2986ad787adc4a50e1 P cc655dbacd6e45deae3b2d906bd63de5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc655dbacd6e45deae3b2d906bd63de5 (127.9.77.129). New cstate: current_term: 1 leader_uuid: "cc655dbacd6e45deae3b2d906bd63de5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08bae345660b4b8286adabf142e8b00e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 40927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc655dbacd6e45deae3b2d906bd63de5" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 44587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f5d02efadb743fba86fc6a20bf3a429" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37805 } health_report { overall_health: UNKNOWN } } }
I20260810 01:48:42.853021 27957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:48:42.846591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57720 (local address 127.9.77.129:44587)
0810 01:48:42.846759 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 01:48:42.846765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:48:42.846783 (+ 18us) server_negotiation.cc:407] Connection header received
0810 01:48:42.846914 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:48:42.846918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:48:42.847017 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:48:42.847162 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:48:42.847764 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:48:42.848642 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:48:42.850076 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:48:42.850442 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:48:42.850554 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:48:42.851089 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:48:42.851938 (+ 849us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:48:42.852514 (+ 576us) server_negotiation.cc:1035] Waiting for connection context
0810 01:48:42.852552 (+ 38us) server_negotiation.cc:299] Negotiation successful
0810 01:48:42.852637 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 01:48:42.863312 27638 raft_consensus.cc:1261] T a6f3fca3751b4b2986ad787adc4a50e1 P 4f5d02efadb743fba86fc6a20bf3a429 [term 1 FOLLOWER]: Refusing update from remote peer cc655dbacd6e45deae3b2d906bd63de5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:48:42.864596 27993 consensus_queue.cc:1035] T a6f3fca3751b4b2986ad787adc4a50e1 P cc655dbacd6e45deae3b2d906bd63de5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f5d02efadb743fba86fc6a20bf3a429" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:48:42.865680 27840 raft_consensus.cc:1261] T a6f3fca3751b4b2986ad787adc4a50e1 P 08bae345660b4b8286adabf142e8b00e [term 1 FOLLOWER]: Refusing update from remote peer cc655dbacd6e45deae3b2d906bd63de5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:48:42.866466 27993 consensus_queue.cc:1035] T a6f3fca3751b4b2986ad787adc4a50e1 P cc655dbacd6e45deae3b2d906bd63de5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08bae345660b4b8286adabf142e8b00e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 40927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:48:42.877213 28003 mvcc.cc:205] Tried to move back new op lower bound from 7316793437635325952 to 7316793437458620416. Current Snapshot: MvccSnapshot[applied={T|T < 7316793437635325952}]
I20260810 01:48:42.886912 28014 mvcc.cc:205] Tried to move back new op lower bound from 7316793437635325952 to 7316793437458620416. Current Snapshot: MvccSnapshot[applied={T|T < 7316793437635325952}]
I20260810 01:48:42.889683 28015 mvcc.cc:205] Tried to move back new op lower bound from 7316793437635325952 to 7316793437458620416. Current Snapshot: MvccSnapshot[applied={T|T < 7316793437635325952}]
I20260810 01:48:42.897929 27187 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:48:42.898212 27187 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:48:42.901954 27187 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be827435f4ba4a0fbc9a735de5e8228e: Sending DeleteTablet for 3 replicas of tablet a6f3fca3751b4b2986ad787adc4a50e1
I20260810 01:48:42.903170 27810 tablet_service.cc:1514] Processing DeleteTablet for tablet a6f3fca3751b4b2986ad787adc4a50e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:48:42 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:34048
I20260810 01:48:42.904760 27386 tablet_service.cc:1514] Processing DeleteTablet for tablet a6f3fca3751b4b2986ad787adc4a50e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:48:42 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:57716
I20260810 01:48:42.906066 27577 tablet_service.cc:1514] Processing DeleteTablet for tablet a6f3fca3751b4b2986ad787adc4a50e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:48:42 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:48486
I20260810 01:48:42.908842 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 27305
I20260810 01:48:42.920080 28030 tablet_replica.cc:330] T a6f3fca3751b4b2986ad787adc4a50e1 P 4f5d02efadb743fba86fc6a20bf3a429: stopping tablet replica
I20260810 01:48:42.920799 28030 raft_consensus.cc:2229] T a6f3fca3751b4b2986ad787adc4a50e1 P 4f5d02efadb743fba86fc6a20bf3a429 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:48:42.921456 28030 raft_consensus.cc:2258] T a6f3fca3751b4b2986ad787adc4a50e1 P 4f5d02efadb743fba86fc6a20bf3a429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:48:42.924314 28030 ts_tablet_manager.cc:1905] T a6f3fca3751b4b2986ad787adc4a50e1 P 4f5d02efadb743fba86fc6a20bf3a429: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:48:42.928225 28028 tablet_replica.cc:330] T a6f3fca3751b4b2986ad787adc4a50e1 P 08bae345660b4b8286adabf142e8b00e: stopping tablet replica
I20260810 01:48:42.928838 28028 raft_consensus.cc:2229] T a6f3fca3751b4b2986ad787adc4a50e1 P 08bae345660b4b8286adabf142e8b00e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:48:42.929355 28028 raft_consensus.cc:2258] T a6f3fca3751b4b2986ad787adc4a50e1 P 08bae345660b4b8286adabf142e8b00e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:48:42.931890 28028 ts_tablet_manager.cc:1905] T a6f3fca3751b4b2986ad787adc4a50e1 P 08bae345660b4b8286adabf142e8b00e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 01:48:42.932870 27162 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.77.129:44587 (error 108)
I20260810 01:48:42.935089 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 27514
I20260810 01:48:42.935118 27953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:48:42.934714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:44587 (local address 127.0.0.1:57734)
0810 01:48:42.934913 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 01:48:42.935000 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.129:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 01:48:42.944636 28028 ts_tablet_manager.cc:1918] T a6f3fca3751b4b2986ad787adc4a50e1 P 08bae345660b4b8286adabf142e8b00e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:48:42.944824 28028 log.cc:1196] T a6f3fca3751b4b2986ad787adc4a50e1 P 08bae345660b4b8286adabf142e8b00e: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786326392877349-9526-0/minicluster-data/ts-2/wal/wals/a6f3fca3751b4b2986ad787adc4a50e1
I20260810 01:48:42.945254 28028 ts_tablet_manager.cc:1939] T a6f3fca3751b4b2986ad787adc4a50e1 P 08bae345660b4b8286adabf142e8b00e: Deleting consensus metadata
W20260810 01:48:42.948146 27162 catalog_manager.cc:4584] TS cc655dbacd6e45deae3b2d906bd63de5 (127.9.77.129:44587): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a6f3fca3751b4b2986ad787adc4a50e1: Network error: Client connection negotiation failed: client connection to 127.9.77.129:44587: connect: Connection refused (error 111)
I20260810 01:48:42.948414 27163 catalog_manager.cc:4841] TS 08bae345660b4b8286adabf142e8b00e (127.9.77.131:40927): tablet a6f3fca3751b4b2986ad787adc4a50e1 (table test-table [id=e0a3799b32d9411baca322790f76949d]) successfully deleted
I20260810 01:48:42.954095 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 27718
I20260810 01:48:42.965381 27953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:48:42.964856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:37805 (local address 127.0.0.1:48492)
0810 01:48:42.965143 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 01:48:42.965224 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.130:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260810 01:48:42.973904 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 27133
I20260810 01:48:42.999166 9526 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 01:48:43.360949 9526 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 01:48:43.361183 9526 mini_ranger.cc:66] Stopping Ranger...
I20260810 01:48:43.717840 9526 mini_ranger.cc:68] Stopped Ranger
2026-08-10 01:48:43.718 UTC [26485] LOG: received smart shutdown request
2026-08-10 01:48:43.722 UTC [26485] LOG: background worker "logical replication launcher" (PID 26506) exited with exit code 1
2026-08-10 01:48:43.723 UTC [26497] LOG: shutting down
2026-08-10 01:48:43.893 UTC [26485] LOG: database system is shut down
2026-08-10T01:48:43Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (55222 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:48:44 dist-test-slave-kvfs krb5kdc[28046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:48:44 dist-test-slave-kvfs krb5kdc[28046](info): set up 2 sockets
Aug 10 01:48:44 dist-test-slave-kvfs krb5kdc[28046](info): commencing operation
krb5kdc: starting...
W20260810 01:48:45.326090 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.272s user 0.000s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:48:45 dist-test-slave-kvfs krb5kdc[28046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:48:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:48:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.77.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.77.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.77.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/rangeradmin_127.9.77.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.77.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/rangeradmin_127.9.77.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.77.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.77.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.77.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/rangerlookup_127.9.77.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.77.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/rangerlookup_127.9.77.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.77.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.77.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.77.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/HTTP_127.9.77.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.77.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/HTTP_127.9.77.148@KRBTEST.COM.keytab.
I20260810 01:48:45.552109 9526 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taski_V3Up/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/postgres -l logfile start
2026-08-10 01:48:48.385 UTC [28091] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 01:48:48.386 UTC [28091] LOG: listening on IPv4 address "127.9.77.148", port 44267
2026-08-10 01:48:48.390 UTC [28091] LOG: listening on Unix socket "/tmp/.s.PGSQL.44267"
2026-08-10 01:48:48.417 UTC [28094] LOG: database system was shut down at 2026-08-10 01:48:46 UTC
2026-08-10 01:48:48.421 UTC [28091] LOG: database system is ready to accept connections
I20260810 01:48:49.620584 9526 mini_postgres.cc:96] Postgres bound to 44267
2026-08-10 01:48:49.630 UTC [28116] FATAL: database "slave" does not exist
127.9.77.148:44267 - accepting connections
I20260810 01:48:49.631328 9526 mini_ranger.cc:161] Starting Ranger...
I20260810 01:48:49.652746 9526 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 01:48:50.218664 9526 mini_ranger.cc:87] Created ranger Postgres database
I20260810 01:48:50.218791 9526 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ranger-admin
2026-08-10 01:48:50,520 [I] DB FLAVOR :POSTGRES
2026-08-10 01:48:50,521 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 01:48:50,521 [I] Checking connection..
2026-08-10 01:48:50,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:44267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 01:48:50,803 [I] Checking connection passed.
2026-08-10 01:48:50,803 [I] --------- Verifying version history table ---------
2026-08-10 01:48:50,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:44267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 01:48:51,077 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 01:48:51,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:44267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 01:48:51,348 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 01:48:51,348 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 01:48:51,348 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:44267/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 01:48:51.633 UTC [28215] WARNING: there is no transaction in progress
2026-08-10 01:48:51,645 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 01:48:51,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:44267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 01:48:51,939 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 01:48:51,940 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 01:48:51,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:44267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 01:48:52,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:44267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 01:48:52,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:44267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-kvfs.c.gcp-upstream.internal','N') ;"
2026-08-10 01:48:52,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:44267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 01:48:53,065 [I] Table x_portal_user does not exist in database ranger
2026-08-10 01:48:53,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:44267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 01:48:53,372 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 01:48:53,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:44267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 01:48:53,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:44267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 01:48:53,954 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 01:48:53,955 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:44267/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 01:48:55.118 UTC [28375] WARNING: there is no transaction in progress
2026-08-10 01:48:55.139 UTC [28375] WARNING: there is no transaction in progress
2026-08-10 01:48:55.159 UTC [28375] WARNING: there is no transaction in progress
2026-08-10 01:48:55.178 UTC [28375] WARNING: there is no transaction in progress
2026-08-10 01:48:55.198 UTC [28375] WARNING: there is no transaction in progress
2026-08-10 01:48:55.217 UTC [28375] WARNING: there is no transaction in progress
2026-08-10 01:48:55.285 UTC [28375] WARNING: there is no transaction in progress
2026-08-10 01:48:55.293 UTC [28375] WARNING: there is no transaction in progress
2026-08-10 01:48:55.310 UTC [28375] WARNING: there is no transaction in progress
2026-08-10 01:48:55.322 UTC [28375] WARNING: there is no transaction in progress
2026-08-10 01:48:55.335 UTC [28375] WARNING: there is no transaction in progress
2026-08-10 01:48:55.351 UTC [28375] WARNING: there is no transaction in progress
2026-08-10 01:48:55.361 UTC [28375] WARNING: there is no transaction in progress
2026-08-10 01:48:55.368 UTC [28375] WARNING: there is no transaction in progress
2026-08-10 01:48:55.376 UTC [28375] WARNING: there is no transaction in progress
2026-08-10 01:48:55.383 UTC [28375] WARNING: there is no transaction in progress
2026-08-10 01:48:56,192 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 01:48:56,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:44267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 01:48:56,463 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 01:48:56,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:44267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 01:48:56,760 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 01:48:56,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:44267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 01:48:57,046 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 01:48:57,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:44267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 01:48:57,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:44267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 01:48:57,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:44267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-kvfs.c.gcp-upstream.internal';"
2026-08-10 01:48:57,880 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 01:48:57,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:44267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 01:48:58,187 [I] DB_PATCHES have already been applied
I20260810 01:48:58.194948 9526 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ranger-admin:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taski_V3Up/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 01:48:58.195078 9526 mini_ranger.cc:193] Using host: 127.9.77.148
I20260810 01:48:58.201210 9526 mini_ranger.cc:227] Ranger admin URL: http://127.9.77.148:32995
Aug 10, 2026 1:48:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 1:48:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 1:48:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 1:48:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 1:48:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 1:48:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 1:48:59 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 01:48:59.634433 9526 mini_ranger.cc:160] Time spent starting Ranger: real 10.003s user 0.001s sys 0.016s
Aug 10, 2026 1:49:21 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 1:49:21 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 10, 2026 1:49:21 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 1:49:21 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 1:49:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 1:49:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 1:49:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 1:49:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 1:49:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 1:49:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 1:49:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 1:49:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 1:49:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 1:49:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 1:49:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 1:49:21 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 1:49:21 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 1:49:23 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 01:49:23.658306 9526 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.77.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.77.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.77.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/rangerkms_127.9.77.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.77.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/rangerkms_127.9.77.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.77.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.77.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.77.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/HTTP_127.9.77.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.77.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/HTTP_127.9.77.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 01:49:23 dist-test-slave-kvfs krb5kdc[28046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326563, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 01:49:23.771474 9526 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 01:49:23.785907 9526 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 01:49:24.781380 9526 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 01:49:24.781503 9526 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ranger-kms
I20260810 01:49:24.781524 9526 mini_ranger_kms.cc:227] Using postgres at 127.9.77.148:44267
2026-08-10 01:49:25,007 [I] DB FLAVOR :POSTGRES
2026-08-10 01:49:25,007 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 01:49:25,007 [I] Checking connection
2026-08-10 01:49:25,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:44267/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 01:49:25,295 [I] connection success
2026-08-10 01:49:25,296 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 01:49:25,296 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 01:49:25,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:44267/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 01:49:25,567 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 01:49:25,567 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 01:49:25,567 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 01:49:25,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.77.148:44267/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 01:49:25,895 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 01:49:26.493831 9526 mini_ranger_kms.cc:310] Created kms service
I20260810 01:49:26.691025 9526 mini_ranger_kms.cc:326] Created kudu user
I20260810 01:49:26.743084 9526 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 01:49:26 dist-test-slave-kvfs krb5kdc[28046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326563, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.77.148@KRBTEST.COM
I20260810 01:49:27.219487 9526 mini_ranger_kms.cc:384] Added ranger policy
I20260810 01:49:27.219712 9526 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ranger-kms:/tmp/dist-test-taski_V3Up/build/debug/bin/postgresql.jar:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taski_V3Up/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 01:49:27.219741 9526 mini_ranger_kms.cc:240] Using host: 127.9.77.148
I20260810 01:49:27.227123 9526 mini_ranger_kms.cc:276] Ranger KMS PID: 28651
I20260810 01:49:27.227247 9526 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.77.148:54701
Aug 10, 2026 1:49:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 1:49:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 1:49:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 1:49:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 1:49:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 1:49:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 01:49:29 dist-test-slave-kvfs krb5kdc[28046](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786326569, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.77.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:49:30.180464 9526 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.409s user 0.011s sys 0.201s
I20260810 01:49:30.180637 9526 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 01:49:30.180719 9526 mini_ranger_kms.cc:401] 127.9.77.148:54701/kms/v1/keys
Aug 10 01:49:30 dist-test-slave-kvfs krb5kdc[28046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326569, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.77.148@KRBTEST.COM for HTTP/127.9.77.148@KRBTEST.COM
Aug 10 01:49:30 dist-test-slave-kvfs krb5kdc[28046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326569, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.77.148@KRBTEST.COM for HTTP/127.9.77.148@KRBTEST.COM
Aug 10 01:49:33 dist-test-slave-kvfs krb5kdc[28046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:49:33.435277 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:33463
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:40393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:33463
--ranger_config_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.77.148:54701/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:49:33.591318 28750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:49:33.591701 28750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:49:33.591744 28750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:49:33.596014 28750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:49:33.596160 28750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:49:33.596181 28750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:49:33.596199 28750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:49:33.596258 28750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:49:33.602497 28750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:40393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.77.148:54701/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:33463
--ranger_config_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:33463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:49:33.604075 28750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:49:33.605504 28750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:49:33.617877 28758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:33.617851 28757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:33.618234 28750 server_base.cc:1034] running on GCE node
W20260810 01:49:33.618484 28761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:33.619227 28750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:49:33.620841 28750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:49:33.622046 28750 hybrid_clock.cc:648] HybridClock initialized: now 1786326573622027 us; error 40 us; skew 500 ppm
Aug 10 01:49:33 dist-test-slave-kvfs krb5kdc[28046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326573, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:49:33.625783 28750 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:49:33.627282 28750 webserver.cc:460] Webserver started at http://127.9.77.190:45443/ using document root <none> and password file <none>
I20260810 01:49:33.628227 28750 fs_manager.cc:359] Metadata directory not provided
I20260810 01:49:33.628312 28750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:49:33.628608 28750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 01:49:33 dist-test-slave-kvfs krb5kdc[28046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326573, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for HTTP/127.9.77.148@KRBTEST.COM
I20260810 01:49:33.685312 28750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "3bac3a48e8874ba88778744fe9eaa9fc"
format_stamp: "Formatted at 2026-08-10 01:49:33 on dist-test-slave-kvfs"
server_key: "2661dea676c559b40f2d12f6c81a047a"
server_key_iv: "2fdfbad7ea7ae5ca789803a1968dc025"
server_key_version: "kuduclusterkey@0"
I20260810 01:49:33.686152 28750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "3bac3a48e8874ba88778744fe9eaa9fc"
format_stamp: "Formatted at 2026-08-10 01:49:33 on dist-test-slave-kvfs"
server_key: "2661dea676c559b40f2d12f6c81a047a"
server_key_iv: "2fdfbad7ea7ae5ca789803a1968dc025"
server_key_version: "kuduclusterkey@0"
I20260810 01:49:33.691229 28750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 01:49:33.710186 28770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:33.712308 28750 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 01:49:33.712514 28750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "3bac3a48e8874ba88778744fe9eaa9fc"
format_stamp: "Formatted at 2026-08-10 01:49:33 on dist-test-slave-kvfs"
server_key: "2661dea676c559b40f2d12f6c81a047a"
server_key_iv: "2fdfbad7ea7ae5ca789803a1968dc025"
server_key_version: "kuduclusterkey@0"
I20260810 01:49:33.712702 28750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:49:33.732122 28750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:49:33.735783 28750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:49:33.736097 28750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:49:33.746024 28750 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:33463
I20260810 01:49:33.746059 28864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:33463 every 8 connection(s)
I20260810 01:49:33.747277 28750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:49:33.748513 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 28750
I20260810 01:49:33.748680 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:49:33.749107 9526 external_mini_cluster.cc:1428] Setting key 0c4bf48c5cef739e250738dce2302e50
I20260810 01:49:33.751825 28865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 01:49:33 dist-test-slave-kvfs krb5kdc[28046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:49:33.760823 28865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc: Bootstrap starting.
I20260810 01:49:33.764382 28865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc: Neither blocks nor log segments found. Creating new log.
I20260810 01:49:33.765441 28871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:33.750925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.9.77.190:33463)
0810 01:49:33.751411 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:33.751439 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:33.751485 (+ 46us) server_negotiation.cc:407] Connection header received
0810 01:49:33.752417 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:33.752446 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:33.752861 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:33.753268 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:33.754231 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:33.755379 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:33.756395 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:33.756931 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:33.757118 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:33.760060 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:33.760071 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:33.760097 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:33.760129 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:33.762020 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:33.762531 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:33.762537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:33.762543 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:33.762665 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:33.762981 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:33.762985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:33.762986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:33.763485 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:33.763716 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:33.764090 (+ 374us) server_negotiation.cc:299] Negotiation successful
0810 01:49:33.764461 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":163,"threads_started":1}
I20260810 01:49:33.766141 28865 log.cc:824] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc: Log is configured to *not* fsync() on all Append() calls
I20260810 01:49:33.769017 28865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc: No bootstrap required, opened a new log
I20260810 01:49:33.772867 28865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bac3a48e8874ba88778744fe9eaa9fc" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33463 } }
I20260810 01:49:33.773190 28865 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:49:33.773258 28865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bac3a48e8874ba88778744fe9eaa9fc, State: Initialized, Role: FOLLOWER
I20260810 01:49:33.773811 28865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bac3a48e8874ba88778744fe9eaa9fc" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33463 } }
I20260810 01:49:33.773959 28865 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:49:33.774048 28865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:49:33.774160 28865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:49:33.775307 28865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bac3a48e8874ba88778744fe9eaa9fc" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33463 } }
I20260810 01:49:33.775739 28865 leader_election.cc:302] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bac3a48e8874ba88778744fe9eaa9fc; no voters:
I20260810 01:49:33.776190 28865 leader_election.cc:288] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:49:33.776324 28876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:49:33.776576 28876 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [term 1 LEADER]: Becoming Leader. State: Replica: 3bac3a48e8874ba88778744fe9eaa9fc, State: Running, Role: LEADER
I20260810 01:49:33.777010 28876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bac3a48e8874ba88778744fe9eaa9fc" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33463 } }
I20260810 01:49:33.777767 28865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:49:33.778566 28876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bac3a48e8874ba88778744fe9eaa9fc. Latest consensus state: current_term: 1 leader_uuid: "3bac3a48e8874ba88778744fe9eaa9fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bac3a48e8874ba88778744fe9eaa9fc" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33463 } } }
I20260810 01:49:33.778781 28876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [sys.catalog]: This master's current role is: LEADER
I20260810 01:49:33.779290 28886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:49:33.779593 28877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bac3a48e8874ba88778744fe9eaa9fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bac3a48e8874ba88778744fe9eaa9fc" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33463 } } }
I20260810 01:49:33.779785 28877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [sys.catalog]: This master's current role is: LEADER
I20260810 01:49:33.783479 28886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:49:33.784071 28865 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 01:49:33.799722 28886 catalog_manager.cc:1347] Generated new cluster ID: 2eec47ad24fc45aa96cef02e225e48ef
I20260810 01:49:33.799861 28886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:49:33.813791 28886 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:49:33.816430 28886 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:49:33.827546 28886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc: Generated new TSK 0
I20260810 01:49:33.828909 28886 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.77.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
I20260810 01:49:34.655364 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.77.148:54701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:0
--local_ip_for_outbound_sockets=127.9.77.129
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:33463
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:40393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:49:34.805569 28927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:49:34.805939 28927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:49:34.806017 28927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:49:34.811123 28927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:49:34.811229 28927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:49:34.811324 28927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:49:34.819694 28927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:40393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.77.148:54701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:33463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:49:34.821398 28927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:49:34.822942 28927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:49:34.838155 28945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:34.838179 28939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:34.838348 28940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:34.839493 28927 server_base.cc:1034] running on GCE node
I20260810 01:49:34.840135 28927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:49:34.841146 28927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:49:34.842363 28927 hybrid_clock.cc:648] HybridClock initialized: now 1786326574842327 us; error 46 us; skew 500 ppm
Aug 10 01:49:34 dist-test-slave-kvfs krb5kdc[28046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326574, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:49:34.845942 28927 init.cc:377] Logged in from keytab as kudu/127.9.77.129@KRBTEST.COM (short username kudu)
I20260810 01:49:34.847226 28927 webserver.cc:460] Webserver started at http://127.9.77.129:35887/ using document root <none> and password file <none>
I20260810 01:49:34.848100 28927 fs_manager.cc:359] Metadata directory not provided
I20260810 01:49:34.848178 28927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:49:34.848461 28927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 01:49:34 dist-test-slave-kvfs krb5kdc[28046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326574, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for HTTP/127.9.77.148@KRBTEST.COM
I20260810 01:49:34.874948 28927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/data/instance:
uuid: "a0057c35515b48d596bcc6cb64f28713"
format_stamp: "Formatted at 2026-08-10 01:49:34 on dist-test-slave-kvfs"
server_key: "346bfb753d535e26b7ee76a9c3a57418"
server_key_iv: "3825978e02918ff00f453c2d743f1e0a"
server_key_version: "kuduclusterkey@0"
I20260810 01:49:34.875937 28927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance:
uuid: "a0057c35515b48d596bcc6cb64f28713"
format_stamp: "Formatted at 2026-08-10 01:49:34 on dist-test-slave-kvfs"
server_key: "346bfb753d535e26b7ee76a9c3a57418"
server_key_iv: "3825978e02918ff00f453c2d743f1e0a"
server_key_version: "kuduclusterkey@0"
I20260810 01:49:34.886852 28927 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.002s sys 0.003s
I20260810 01:49:34.916332 28967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:34.928694 28927 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.002s sys 0.003s
I20260810 01:49:34.929191 28927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "a0057c35515b48d596bcc6cb64f28713"
format_stamp: "Formatted at 2026-08-10 01:49:34 on dist-test-slave-kvfs"
server_key: "346bfb753d535e26b7ee76a9c3a57418"
server_key_iv: "3825978e02918ff00f453c2d743f1e0a"
server_key_version: "kuduclusterkey@0"
I20260810 01:49:34.929415 28927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:49:34.964471 28927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:49:34.968477 28927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:49:34.968732 28927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:49:34.969534 28927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:49:34.970696 28927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:49:34.970772 28927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:34.970867 28927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:49:34.970914 28927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:34.984496 28927 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:42371
I20260810 01:49:34.986914 28927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:49:34.988212 29164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:42371 every 8 connection(s)
I20260810 01:49:34.991467 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 28927
I20260810 01:49:34.991619 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance
I20260810 01:49:34.992046 9526 external_mini_cluster.cc:1428] Setting key 1e41d15f1779740c9dc45c83e98f5e32
Aug 10 01:49:35 dist-test-slave-kvfs krb5kdc[28046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326574, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.77.130@KRBTEST.COM; defaulting to no policy
I20260810 01:49:35.010955 29167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:34.993127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:46251 (local address 127.9.77.190:33463)
0810 01:49:34.993464 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:34.993469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:34.994515 (+ 1046us) server_negotiation.cc:407] Connection header received
0810 01:49:34.996041 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:34.996046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:34.996111 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:34.996203 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:34.998045 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:34.999024 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:35.000947 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:35.001330 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:35.001462 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:35.005429 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:35.005433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:35.005437 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:35.005474 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:35.007763 (+ 2289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:35.008554 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:35.008560 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:35.008563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:35.008679 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:35.009205 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:35.009209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:35.009211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:35.009578 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:35.009692 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:35.010544 (+ 852us) server_negotiation.cc:299] Negotiation successful
0810 01:49:35.010722 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":116,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.130@KRBTEST.COM" created.
I20260810 01:49:35.013200 29168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:34.993730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33463 (local address 127.9.77.129:46251)
0810 01:49:34.994339 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:34.994399 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:34.995628 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:34.996591 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:34.996602 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:34.997033 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:49:34.997858 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:34.997879 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:34.999185 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:34.999189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:35.000788 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:35.000800 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:35.001697 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:35.001701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:35.002024 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:35.002921 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:49:35.002947 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:49:35.005163 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:49:35.007935 (+ 2772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:35.007943 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:35.007968 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:35.008414 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:35.008810 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:35.008814 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:35.008817 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:35.009055 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:35.009718 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:49:35.009727 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:49:35.010173 (+ 446us) client_negotiation.cc:769] Sending connection context
0810 01:49:35.010452 (+ 279us) client_negotiation.cc:240] Negotiation successful
0810 01:49:35.011053 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":205,"threads_started":1}
Aug 10 01:49:35 dist-test-slave-kvfs krb5kdc[28046](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786326575, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:49:35.016486 29165 heartbeater.cc:347] Connected to a master server at 127.9.77.190:33463
I20260810 01:49:35.016870 29165 heartbeater.cc:464] Registering TS with master...
I20260810 01:49:35.017661 29165 heartbeater.cc:511] Master 127.9.77.190:33463 requested a full tablet report, sending...
I20260810 01:49:35.020146 28796 ts_manager.cc:194] Registered new tserver with Master: a0057c35515b48d596bcc6cb64f28713 (127.9.77.129:42371)
I20260810 01:49:35.022002 28796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.129@KRBTEST.COM'} at 127.9.77.129:46251
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
I20260810 01:49:35.036265 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.77.148:54701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:0
--local_ip_for_outbound_sockets=127.9.77.130
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:33463
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:40393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:49:35.211385 29171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:49:35.211851 29171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:49:35.211930 29171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:49:35.218417 29171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:49:35.218524 29171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:49:35.218621 29171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:49:35.227566 29171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:40393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.77.148:54701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:33463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:49:35.229665 29171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:49:35.231483 29171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:49:35.247351 29179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:35.248131 29181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:35.248525 29178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:35.252470 29171 server_base.cc:1034] running on GCE node
I20260810 01:49:35.253255 29171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:49:35.254364 29171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:49:35.257678 29171 hybrid_clock.cc:648] HybridClock initialized: now 1786326575257551 us; error 135 us; skew 500 ppm
Aug 10 01:49:35 dist-test-slave-kvfs krb5kdc[28046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326575, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:49:35.262271 29171 init.cc:377] Logged in from keytab as kudu/127.9.77.130@KRBTEST.COM (short username kudu)
I20260810 01:49:35.263800 29171 webserver.cc:460] Webserver started at http://127.9.77.130:43173/ using document root <none> and password file <none>
I20260810 01:49:35.264758 29171 fs_manager.cc:359] Metadata directory not provided
I20260810 01:49:35.264874 29171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:49:35.265228 29171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 01:49:35 dist-test-slave-kvfs krb5kdc[28046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326575, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for HTTP/127.9.77.148@KRBTEST.COM
I20260810 01:49:35.286990 29171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/data/instance:
uuid: "d9c68f54081441939d921df17f8c285e"
format_stamp: "Formatted at 2026-08-10 01:49:35 on dist-test-slave-kvfs"
server_key: "283d62fb530bd183d628a998e6114e21"
server_key_iv: "678219c6bded061764edf08b29bbbcbc"
server_key_version: "kuduclusterkey@0"
I20260810 01:49:35.287937 29171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance:
uuid: "d9c68f54081441939d921df17f8c285e"
format_stamp: "Formatted at 2026-08-10 01:49:35 on dist-test-slave-kvfs"
server_key: "283d62fb530bd183d628a998e6114e21"
server_key_iv: "678219c6bded061764edf08b29bbbcbc"
server_key_version: "kuduclusterkey@0"
I20260810 01:49:35.294262 29171 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260810 01:49:35.317466 29193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:35.320163 29171 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260810 01:49:35.320463 29171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "d9c68f54081441939d921df17f8c285e"
format_stamp: "Formatted at 2026-08-10 01:49:35 on dist-test-slave-kvfs"
server_key: "283d62fb530bd183d628a998e6114e21"
server_key_iv: "678219c6bded061764edf08b29bbbcbc"
server_key_version: "kuduclusterkey@0"
I20260810 01:49:35.320760 29171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:49:35.354961 29171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:49:35.359727 29171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:49:35.360095 29171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:49:35.361127 29171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:49:35.362519 29171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:49:35.362644 29171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 01:49:35.362780 29171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:49:35.362854 29171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:35.379227 29171 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:38413
I20260810 01:49:35.380926 29171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:49:35.382913 29370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:38413 every 8 connection(s)
I20260810 01:49:35.383572 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 29171
I20260810 01:49:35.383738 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance
I20260810 01:49:35.384174 9526 external_mini_cluster.cc:1428] Setting key 021748d17921fba9fc0283b2cc3b640b
WARNING: no policy specified for kudu/127.9.77.131@KRBTEST.COM; defaulting to no policy
Aug 10 01:49:35 dist-test-slave-kvfs krb5kdc[28046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326575, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:49:35.422629 29167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:35.395100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:47177 (local address 127.9.77.190:33463)
0810 01:49:35.395294 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:35.395301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:35.396634 (+ 1333us) server_negotiation.cc:407] Connection header received
0810 01:49:35.398141 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:35.398146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:35.398219 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:35.398301 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:35.403576 (+ 5275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:35.404550 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:35.407834 (+ 3284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:35.408237 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:35.408346 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:35.417216 (+ 8870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:35.417222 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:35.417225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:35.417280 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:35.419438 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:35.420318 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:35.420323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:35.420325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:35.420433 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:35.420934 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:35.420938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:35.420939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:35.421256 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:35.421357 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:35.422232 (+ 875us) server_negotiation.cc:299] Negotiation successful
0810 01:49:35.422417 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 01:49:35.425266 29375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:35.393667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33463 (local address 127.9.77.130:47177)
0810 01:49:35.396428 (+ 2761us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:35.396503 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:35.397833 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:35.398855 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:35.398866 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:35.399343 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:49:35.400236 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:35.400271 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:35.404748 (+ 4477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:35.404752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:35.406409 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:35.406422 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:35.408620 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:35.408625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:35.408880 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:35.414623 (+ 5743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:49:35.414656 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:49:35.416925 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:49:35.419633 (+ 2708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:35.419641 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:35.419668 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:35.420165 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:35.420558 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:35.420561 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:35.420564 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:35.420807 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:35.421433 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:49:35.421441 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:49:35.421893 (+ 452us) client_negotiation.cc:769] Sending connection context
0810 01:49:35.423231 (+ 1338us) client_negotiation.cc:240] Negotiation successful
0810 01:49:35.423733 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2459,"thread_start_us":200,"threads_started":1}
I20260810 01:49:35.427969 29371 heartbeater.cc:347] Connected to a master server at 127.9.77.190:33463
I20260810 01:49:35.428361 29371 heartbeater.cc:464] Registering TS with master...
I20260810 01:49:35.429137 29371 heartbeater.cc:511] Master 127.9.77.190:33463 requested a full tablet report, sending...
I20260810 01:49:35.430795 28796 ts_manager.cc:194] Registered new tserver with Master: d9c68f54081441939d921df17f8c285e (127.9.77.130:38413)
I20260810 01:49:35.431708 28796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.130@KRBTEST.COM'} at 127.9.77.130:47177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
I20260810 01:49:35.473192 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.77.148:54701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:0
--local_ip_for_outbound_sockets=127.9.77.131
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:33463
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:40393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 01:49:35 dist-test-slave-kvfs krb5kdc[28046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326575, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for HTTP/127.9.77.148@KRBTEST.COM
W20260810 01:49:35.677183 29378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:49:35.677650 29378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:49:35.677726 29378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:49:35.684365 29378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:49:35.684580 29378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:49:35.684897 29378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:49:35.693913 29378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:40393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.77.148:54701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:33463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:49:35.696262 29378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:49:35.698091 29378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:49:35.714318 29386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:35.715698 29378 server_base.cc:1034] running on GCE node
W20260810 01:49:35.716255 29388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:35.720191 29384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:35.720999 29378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:49:35.722182 29378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:49:35.725471 29378 hybrid_clock.cc:648] HybridClock initialized: now 1786326575725366 us; error 114 us; skew 500 ppm
Aug 10 01:49:35 dist-test-slave-kvfs krb5kdc[28046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326575, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:49:35.730512 29378 init.cc:377] Logged in from keytab as kudu/127.9.77.131@KRBTEST.COM (short username kudu)
I20260810 01:49:35.732378 29378 webserver.cc:460] Webserver started at http://127.9.77.131:44411/ using document root <none> and password file <none>
I20260810 01:49:35.733392 29378 fs_manager.cc:359] Metadata directory not provided
I20260810 01:49:35.733567 29378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:49:35.733971 29378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 01:49:35 dist-test-slave-kvfs krb5kdc[28046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326575, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for HTTP/127.9.77.148@KRBTEST.COM
I20260810 01:49:35.763861 29378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/data/instance:
uuid: "839eeb6e78994a20985097a9f62737c4"
format_stamp: "Formatted at 2026-08-10 01:49:35 on dist-test-slave-kvfs"
server_key: "cf5d600c0f4917c22262010cdc57b27d"
server_key_iv: "1f5d01c35cb79d73757d4fa319298ec5"
server_key_version: "kuduclusterkey@0"
I20260810 01:49:35.766652 29378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance:
uuid: "839eeb6e78994a20985097a9f62737c4"
format_stamp: "Formatted at 2026-08-10 01:49:35 on dist-test-slave-kvfs"
server_key: "cf5d600c0f4917c22262010cdc57b27d"
server_key_iv: "1f5d01c35cb79d73757d4fa319298ec5"
server_key_version: "kuduclusterkey@0"
I20260810 01:49:35.780750 29378 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.006s sys 0.000s
I20260810 01:49:35.804867 29400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:35.807034 29378 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260810 01:49:35.807209 29378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "839eeb6e78994a20985097a9f62737c4"
format_stamp: "Formatted at 2026-08-10 01:49:35 on dist-test-slave-kvfs"
server_key: "cf5d600c0f4917c22262010cdc57b27d"
server_key_iv: "1f5d01c35cb79d73757d4fa319298ec5"
server_key_version: "kuduclusterkey@0"
I20260810 01:49:35.807399 29378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:49:35.825008 29378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:49:35.829260 29378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:49:35.829488 29378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:49:35.830327 29378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:49:35.831558 29378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:49:35.831629 29378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:35.831696 29378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:49:35.831732 29378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:35.848349 29378 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:41809
I20260810 01:49:35.849779 29378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:49:35.853168 29579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:41809 every 8 connection(s)
I20260810 01:49:35.855496 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 29378
I20260810 01:49:35.855639 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance
I20260810 01:49:35.855998 9526 external_mini_cluster.cc:1428] Setting key e5774a2625633de808482b26f67d9857
Aug 10 01:49:35 dist-test-slave-kvfs krb5kdc[28046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326575, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:49:35.876639 29167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:35.856643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:48137 (local address 127.9.77.190:33463)
0810 01:49:35.856804 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:35.856809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:35.858794 (+ 1985us) server_negotiation.cc:407] Connection header received
0810 01:49:35.860286 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:35.860290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:35.860356 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:35.860409 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:35.862520 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:35.863389 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:35.865533 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:35.865998 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:35.866080 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:35.871677 (+ 5597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:35.871681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:35.871685 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:35.871731 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:35.873690 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:35.874445 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:35.874449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:35.874452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:35.874549 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:35.875016 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:35.875019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:35.875021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:35.875291 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:35.875373 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:35.876274 (+ 901us) server_negotiation.cc:299] Negotiation successful
0810 01:49:35.876442 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 01:49:35.879021 29584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:35.854000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33463 (local address 127.9.77.131:48137)
0810 01:49:35.858584 (+ 4584us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:35.858662 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:35.859960 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:35.860972 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:35.860984 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:35.861465 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:49:35.862327 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:35.862349 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:35.863586 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:35.863590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:35.865329 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:35.865348 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:35.866367 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:35.866371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:35.866634 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:35.867540 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:49:35.867566 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:49:35.871457 (+ 3891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:49:35.873858 (+ 2401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:35.873866 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:35.873893 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:35.874325 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:35.874655 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:35.874659 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:35.874661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:35.874897 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:35.875442 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:49:35.875450 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:49:35.875906 (+ 456us) client_negotiation.cc:769] Sending connection context
0810 01:49:35.876951 (+ 1045us) client_negotiation.cc:240] Negotiation successful
0810 01:49:35.877476 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4235,"thread_start_us":203,"threads_started":1}
I20260810 01:49:35.880821 29580 heartbeater.cc:347] Connected to a master server at 127.9.77.190:33463
I20260810 01:49:35.881168 29580 heartbeater.cc:464] Registering TS with master...
I20260810 01:49:35.881930 29580 heartbeater.cc:511] Master 127.9.77.190:33463 requested a full tablet report, sending...
I20260810 01:49:35.885051 28796 ts_manager.cc:194] Registered new tserver with Master: 839eeb6e78994a20985097a9f62737c4 (127.9.77.131:41809)
I20260810 01:49:35.886045 28796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.131@KRBTEST.COM'} at 127.9.77.131:48137
I20260810 01:49:35.898780 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 01:49:35.899813 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 28927
I20260810 01:49:36.030179 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 29171
I20260810 01:49:36.069788 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 29378
Aug 10 01:49:36 dist-test-slave-kvfs krb5kdc[28046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326575, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for HTTP/127.9.77.148@KRBTEST.COM
I20260810 01:49:36.103449 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 28750
I20260810 01:49:36.127190 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:33463
--webserver_interface=127.9.77.190
--webserver_port=45443
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:40393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:33463
--ranger_config_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.77.148:54701/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.77.148:54701/kms with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:49:36.292618 29587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:49:36.292964 29587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:49:36.293007 29587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:49:36.297210 29587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:49:36.297286 29587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:49:36.297307 29587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:49:36.297323 29587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:49:36.297338 29587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:49:36.303306 29587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:40393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.77.148:54701/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:33463
--ranger_config_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:33463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=45443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:49:36.304821 29587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:49:36.306135 29587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:49:36.316843 29595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:36.317056 29600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:36.316905 29596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:36.316969 29587 server_base.cc:1034] running on GCE node
I20260810 01:49:36.317894 29587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:49:36.319306 29587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:49:36.320472 29587 hybrid_clock.cc:648] HybridClock initialized: now 1786326576320442 us; error 48 us; skew 500 ppm
Aug 10 01:49:36 dist-test-slave-kvfs krb5kdc[28046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326576, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:49:36.323772 29587 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:49:36.324986 29587 webserver.cc:460] Webserver started at http://127.9.77.190:45443/ using document root <none> and password file <none>
I20260810 01:49:36.325666 29587 fs_manager.cc:359] Metadata directory not provided
I20260810 01:49:36.325766 29587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 01:49:36 dist-test-slave-kvfs krb5kdc[28046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326576, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for HTTP/127.9.77.148@KRBTEST.COM
I20260810 01:49:36.421491 29587 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 01:49:36.424718 29611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:36.426565 29587 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 01:49:36.426757 29587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "3bac3a48e8874ba88778744fe9eaa9fc"
format_stamp: "Formatted at 2026-08-10 01:49:33 on dist-test-slave-kvfs"
server_key: "2661dea676c559b40f2d12f6c81a047a"
server_key_iv: "2fdfbad7ea7ae5ca789803a1968dc025"
server_key_version: "kuduclusterkey@0"
I20260810 01:49:36.427513 29587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:49:36.442864 29587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:49:36.446580 29587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:49:36.446875 29587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:49:36.456308 29587 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:33463
I20260810 01:49:36.456343 29705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:33463 every 8 connection(s)
I20260810 01:49:36.457733 29587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:49:36.463400 29706 sys_catalog.cc:263] Verifying existing consensus state
I20260810 01:49:36.464524 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 29587
I20260810 01:49:36.465065 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.77.148:54701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:42371
--local_ip_for_outbound_sockets=127.9.77.129
--tserver_master_addrs=127.9.77.190:33463
--webserver_port=35887
--webserver_interface=127.9.77.129
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:40393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.77.148:54701/kms with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 01:49:36.465749 29706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc: Bootstrap starting.
I20260810 01:49:36.486037 29706 log.cc:824] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc: Log is configured to *not* fsync() on all Append() calls
I20260810 01:49:36.494455 29706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 01:49:36.494980 29706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc: Bootstrap complete.
I20260810 01:49:36.499974 29706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bac3a48e8874ba88778744fe9eaa9fc" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33463 } }
I20260810 01:49:36.500566 29706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bac3a48e8874ba88778744fe9eaa9fc, State: Initialized, Role: FOLLOWER
I20260810 01:49:36.501178 29706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bac3a48e8874ba88778744fe9eaa9fc" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33463 } }
I20260810 01:49:36.501360 29706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:49:36.501451 29706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:49:36.501585 29706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [term 1 FOLLOWER]: Advancing to term 2
I20260810 01:49:36.504012 29706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bac3a48e8874ba88778744fe9eaa9fc" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33463 } }
I20260810 01:49:36.504500 29706 leader_election.cc:302] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bac3a48e8874ba88778744fe9eaa9fc; no voters:
I20260810 01:49:36.504917 29706 leader_election.cc:288] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 01:49:36.505034 29711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [term 2 FOLLOWER]: Leader election won for term 2
I20260810 01:49:36.505240 29711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [term 2 LEADER]: Becoming Leader. State: Replica: 3bac3a48e8874ba88778744fe9eaa9fc, State: Running, Role: LEADER
I20260810 01:49:36.505589 29711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bac3a48e8874ba88778744fe9eaa9fc" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33463 } }
I20260810 01:49:36.506500 29706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:49:36.507205 29712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3bac3a48e8874ba88778744fe9eaa9fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bac3a48e8874ba88778744fe9eaa9fc" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33463 } } }
I20260810 01:49:36.507351 29712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [sys.catalog]: This master's current role is: LEADER
I20260810 01:49:36.507221 29713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bac3a48e8874ba88778744fe9eaa9fc. Latest consensus state: current_term: 2 leader_uuid: "3bac3a48e8874ba88778744fe9eaa9fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bac3a48e8874ba88778744fe9eaa9fc" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33463 } } }
I20260810 01:49:36.507788 29713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc [sys.catalog]: This master's current role is: LEADER
I20260810 01:49:36.508735 29715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:49:36.512918 29715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:49:36.513554 29706 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 01:49:36.513815 29715 catalog_manager.cc:1259] Loaded cluster ID: 2eec47ad24fc45aa96cef02e225e48ef
I20260810 01:49:36.513926 29715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:49:36.519619 29715 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:49:36.521363 29715 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc: Loaded TSK: 0
I20260810 01:49:36.523599 29715 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 01:49:36.645502 29709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:49:36.646015 29709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:49:36.646088 29709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:49:36.652844 29709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:49:36.652981 29709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:49:36.653087 29709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:49:36.662205 29709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:40393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.77.148:54701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:42371
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=35887
--tserver_master_addrs=127.9.77.190:33463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:49:36.664167 29709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:49:36.666006 29709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:49:36.682327 29765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:36.682303 29766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:36.682780 29768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:36.684069 29709 server_base.cc:1034] running on GCE node
I20260810 01:49:36.684753 29709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:49:36.685834 29709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:49:36.687062 29709 hybrid_clock.cc:648] HybridClock initialized: now 1786326576687012 us; error 55 us; skew 500 ppm
Aug 10 01:49:36 dist-test-slave-kvfs krb5kdc[28046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326576, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:49:36.690793 29709 init.cc:377] Logged in from keytab as kudu/127.9.77.129@KRBTEST.COM (short username kudu)
I20260810 01:49:36.691980 29709 webserver.cc:460] Webserver started at http://127.9.77.129:35887/ using document root <none> and password file <none>
I20260810 01:49:36.692932 29709 fs_manager.cc:359] Metadata directory not provided
I20260810 01:49:36.693056 29709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 01:49:36 dist-test-slave-kvfs krb5kdc[28046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326576, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for HTTP/127.9.77.148@KRBTEST.COM
I20260810 01:49:36.717541 29709 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 01:49:36.722795 29780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:36.724907 29709 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260810 01:49:36.725078 29709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "a0057c35515b48d596bcc6cb64f28713"
format_stamp: "Formatted at 2026-08-10 01:49:34 on dist-test-slave-kvfs"
server_key: "346bfb753d535e26b7ee76a9c3a57418"
server_key_iv: "3825978e02918ff00f453c2d743f1e0a"
server_key_version: "kuduclusterkey@0"
I20260810 01:49:36.726028 29709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:49:36.751981 29709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:49:36.756687 29709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:49:36.756958 29709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:49:36.757915 29709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:49:36.759261 29709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:49:36.759408 29709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:36.759577 29709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:49:36.759685 29709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:36.774842 29709 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:42371
I20260810 01:49:36.774920 29945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:42371 every 8 connection(s)
I20260810 01:49:36.776682 29709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:49:36.780848 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 29709
I20260810 01:49:36.781484 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.77.148:54701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:38413
--local_ip_for_outbound_sockets=127.9.77.130
--tserver_master_addrs=127.9.77.190:33463
--webserver_port=43173
--webserver_interface=127.9.77.130
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:40393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.77.148:54701/kms with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 01:49:36 dist-test-slave-kvfs krb5kdc[28046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326576, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:49:36.812335 29949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:36.782596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33463 (local address 127.9.77.129:43065)
0810 01:49:36.788481 (+ 5885us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:36.788573 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:36.789961 (+ 1388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:36.794120 (+ 4159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:36.794133 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:36.794605 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:49:36.795542 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:36.795568 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:36.798161 (+ 2593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:36.798165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:36.799729 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:36.799741 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:36.800934 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:36.800939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:36.801178 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:36.802234 (+ 1056us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:49:36.802261 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:49:36.804377 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:49:36.807565 (+ 3188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:36.807574 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:36.807592 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:36.808069 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:36.808565 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:36.808568 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:36.808570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:36.808768 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:36.809677 (+ 909us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:49:36.809687 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:49:36.810102 (+ 415us) client_negotiation.cc:769] Sending connection context
0810 01:49:36.810361 (+ 259us) client_negotiation.cc:240] Negotiation successful
0810 01:49:36.810789 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5560,"thread_start_us":194,"threads_started":1}
I20260810 01:49:36.815392 29951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:36.784575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:43065 (local address 127.9.77.190:33463)
0810 01:49:36.792398 (+ 7823us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:36.792429 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:36.792500 (+ 71us) server_negotiation.cc:407] Connection header received
0810 01:49:36.792914 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:36.792935 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:36.793508 (+ 573us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:36.795854 (+ 2346us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:36.796270 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:36.797975 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:36.799906 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:36.800712 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:36.801708 (+ 996us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:36.804659 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:36.804669 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:36.804689 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:36.804727 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:36.807390 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:36.808238 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:36.808245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:36.808251 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:36.808442 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:36.808893 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:36.808897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:36.808898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:36.809502 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:36.813675 (+ 4173us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:36.813866 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 01:49:36.814227 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7580,"thread_start_us":133,"threads_started":1}
I20260810 01:49:36.819120 29946 heartbeater.cc:347] Connected to a master server at 127.9.77.190:33463
I20260810 01:49:36.819459 29946 heartbeater.cc:464] Registering TS with master...
I20260810 01:49:36.820267 29946 heartbeater.cc:511] Master 127.9.77.190:33463 requested a full tablet report, sending...
W20260810 01:49:36.823276 29946 heartbeater.cc:644] Failed to heartbeat to 127.9.77.190:33463 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 01:49:36.823443 29946 heartbeater.cc:464] Registering TS with master...
I20260810 01:49:36.823680 29946 heartbeater.cc:511] Master 127.9.77.190:33463 requested a full tablet report, sending...
I20260810 01:49:36.824753 29946 heartbeater.cc:464] Registering TS with master...
I20260810 01:49:36.824944 29946 heartbeater.cc:511] Master 127.9.77.190:33463 requested a full tablet report, sending...
W20260810 01:49:36.825798 29946 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 01:49:36.991227 29950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:49:36.991815 29950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:49:36.991933 29950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:49:36.998558 29950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:49:36.998715 29950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:49:36.998865 29950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:49:37.008172 29950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:40393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.77.148:54701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:38413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=43173
--tserver_master_addrs=127.9.77.190:33463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:49:37.010224 29950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:49:37.012169 29950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:49:37.028592 29965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:37.030232 29950 server_base.cc:1034] running on GCE node
W20260810 01:49:37.035096 29962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:37.035674 29961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:37.036566 29950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:49:37.037760 29950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:49:37.045288 29950 hybrid_clock.cc:648] HybridClock initialized: now 1786326577045239 us; error 92 us; skew 500 ppm
Aug 10 01:49:37 dist-test-slave-kvfs krb5kdc[28046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326577, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:49:37.050110 29950 init.cc:377] Logged in from keytab as kudu/127.9.77.130@KRBTEST.COM (short username kudu)
I20260810 01:49:37.051756 29950 webserver.cc:460] Webserver started at http://127.9.77.130:43173/ using document root <none> and password file <none>
I20260810 01:49:37.052870 29950 fs_manager.cc:359] Metadata directory not provided
I20260810 01:49:37.053085 29950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 01:49:37 dist-test-slave-kvfs krb5kdc[28046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326577, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for HTTP/127.9.77.148@KRBTEST.COM
I20260810 01:49:37.079017 29950 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 01:49:37.083117 29979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:37.085229 29950 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 01:49:37.085415 29950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "d9c68f54081441939d921df17f8c285e"
format_stamp: "Formatted at 2026-08-10 01:49:35 on dist-test-slave-kvfs"
server_key: "283d62fb530bd183d628a998e6114e21"
server_key_iv: "678219c6bded061764edf08b29bbbcbc"
server_key_version: "kuduclusterkey@0"
I20260810 01:49:37.086337 29950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:49:37.103663 29950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:49:37.107935 29950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:49:37.108222 29950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:49:37.109129 29950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:49:37.110350 29950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:49:37.110467 29950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:37.110555 29950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:49:37.110606 29950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:37.125329 29950 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:38413
I20260810 01:49:37.127334 29950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:49:37.128610 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 29950
I20260810 01:49:37.128636 30142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:38413 every 8 connection(s)
I20260810 01:49:37.129277 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.77.148:54701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:41809
--local_ip_for_outbound_sockets=127.9.77.131
--tserver_master_addrs=127.9.77.190:33463
--webserver_port=44411
--webserver_interface=127.9.77.131
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:40393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.77.148:54701/kms with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 01:49:37 dist-test-slave-kvfs krb5kdc[28046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326577, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:49:37.147686 29951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:37.133630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:43107 (local address 127.9.77.190:33463)
0810 01:49:37.133798 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:37.133804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:37.133958 (+ 154us) server_negotiation.cc:407] Connection header received
0810 01:49:37.135242 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:37.135246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:37.135329 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:37.135434 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:37.137150 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:37.137952 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:37.139345 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:37.139627 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:37.139754 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:37.143074 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:37.143078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:37.143080 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:37.143114 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:37.145081 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:37.145697 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:37.145701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:37.145703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:37.145801 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:37.146224 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:37.146228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:37.146230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:37.146466 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:37.146595 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:37.147239 (+ 644us) server_negotiation.cc:299] Negotiation successful
0810 01:49:37.147431 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 01:49:37.148994 30154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:37.132838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33463 (local address 127.9.77.130:43107)
0810 01:49:37.133590 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:37.133658 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:37.134958 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:37.135683 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:37.135691 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:37.136200 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:49:37.136967 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:37.136986 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:37.138104 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:37.138108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:37.139194 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:37.139205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:37.139825 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:37.139830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:37.140117 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:37.140997 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:49:37.141022 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:49:37.142842 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:49:37.145221 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:37.145230 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:37.145252 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:37.145573 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:37.145920 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:37.145924 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:37.145926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:37.146107 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:37.146591 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:49:37.146600 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:49:37.146939 (+ 339us) client_negotiation.cc:769] Sending connection context
0810 01:49:37.147173 (+ 234us) client_negotiation.cc:240] Negotiation successful
0810 01:49:37.147606 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":218,"threads_started":1}
I20260810 01:49:37.150558 30143 heartbeater.cc:347] Connected to a master server at 127.9.77.190:33463
I20260810 01:49:37.150877 30143 heartbeater.cc:464] Registering TS with master...
I20260810 01:49:37.151733 30143 heartbeater.cc:511] Master 127.9.77.190:33463 requested a full tablet report, sending...
W20260810 01:49:37.153805 30143 heartbeater.cc:644] Failed to heartbeat to 127.9.77.190:33463 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 01:49:37.153993 30143 heartbeater.cc:464] Registering TS with master...
I20260810 01:49:37.154206 30143 heartbeater.cc:511] Master 127.9.77.190:33463 requested a full tablet report, sending...
I20260810 01:49:37.155155 30143 heartbeater.cc:464] Registering TS with master...
I20260810 01:49:37.155370 30143 heartbeater.cc:511] Master 127.9.77.190:33463 requested a full tablet report, sending...
W20260810 01:49:37.156239 30143 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 01:49:37.339593 30153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:49:37.340175 30153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:49:37.340253 30153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:49:37.346946 30153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:49:37.347110 30153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:49:37.347234 30153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:49:37.356128 30153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:40393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.77.148:54701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:41809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=44411
--tserver_master_addrs=127.9.77.190:33463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:49:37.358269 30153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:49:37.360111 30153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:49:37.375610 30163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:37.375809 30161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:37.375610 30160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:37.377383 30153 server_base.cc:1034] running on GCE node
I20260810 01:49:37.378222 30153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:49:37.379324 30153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:49:37.382647 30153 hybrid_clock.cc:648] HybridClock initialized: now 1786326577382596 us; error 68 us; skew 500 ppm
Aug 10 01:49:37 dist-test-slave-kvfs krb5kdc[28046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326577, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:49:37.386488 30153 init.cc:377] Logged in from keytab as kudu/127.9.77.131@KRBTEST.COM (short username kudu)
I20260810 01:49:37.388063 30153 webserver.cc:460] Webserver started at http://127.9.77.131:44411/ using document root <none> and password file <none>
I20260810 01:49:37.388988 30153 fs_manager.cc:359] Metadata directory not provided
I20260810 01:49:37.389137 30153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 01:49:37 dist-test-slave-kvfs krb5kdc[28046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326577, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for HTTP/127.9.77.148@KRBTEST.COM
I20260810 01:49:37.412993 30153 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260810 01:49:37.416620 30180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:37.418642 30153 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 01:49:37.418797 30153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "839eeb6e78994a20985097a9f62737c4"
format_stamp: "Formatted at 2026-08-10 01:49:35 on dist-test-slave-kvfs"
server_key: "cf5d600c0f4917c22262010cdc57b27d"
server_key_iv: "1f5d01c35cb79d73757d4fa319298ec5"
server_key_version: "kuduclusterkey@0"
I20260810 01:49:37.419682 30153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:49:37.440522 30153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:49:37.445010 30153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:49:37.445322 30153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:49:37.446280 30153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:49:37.447690 30153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:49:37.447765 30153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:37.447834 30153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:49:37.447870 30153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:37.461162 30153 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:41809
I20260810 01:49:37.462739 30153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:49:37.463084 30373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:41809 every 8 connection(s)
I20260810 01:49:37.469446 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 30153
Aug 10 01:49:37 dist-test-slave-kvfs krb5kdc[28046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326577, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:49:37.483517 29951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:37.465677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:51697 (local address 127.9.77.190:33463)
0810 01:49:37.465836 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:37.465841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:37.467191 (+ 1350us) server_negotiation.cc:407] Connection header received
0810 01:49:37.468630 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:37.468633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:37.468699 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:37.468788 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:37.471308 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:37.472243 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:37.474041 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:37.474407 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:37.474478 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:37.477897 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:37.477901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:37.477904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:37.477933 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:37.479624 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:37.481240 (+ 1616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:37.481243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:37.481245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:37.481336 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:37.481858 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:37.481861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:37.481863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:37.482163 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:37.482282 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:37.483121 (+ 839us) server_negotiation.cc:299] Negotiation successful
0810 01:49:37.483304 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":56,"spinlock_wait_cycles":1408}
I20260810 01:49:37.483825 30377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:37.470259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58812 (local address 127.9.77.190:33463)
0810 01:49:37.471034 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:37.471044 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:37.471058 (+ 14us) server_negotiation.cc:407] Connection header received
0810 01:49:37.471098 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:37.471101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:37.471149 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:37.471246 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:37.472360 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:37.473111 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:37.474325 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:37.474629 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:37.474714 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:37.477918 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:37.477921 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:37.477923 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:37.477940 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:37.480888 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:37.481453 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:37.481458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:37.481460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:37.481548 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:37.481858 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:37.481861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:37.481863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:37.482421 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:37.482610 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:37.483423 (+ 813us) server_negotiation.cc:299] Negotiation successful
0810 01:49:37.483566 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":683,"spinlock_wait_cycles":3328,"thread_start_us":133,"threads_started":1}
I20260810 01:49:37.485926 30376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:37.466322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:33463 (local address 127.9.77.131:51697)
0810 01:49:37.467003 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:37.467066 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:37.468378 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:37.469162 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:37.469172 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:37.469613 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:49:37.471131 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:37.471157 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:37.472426 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:37.472429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:37.473901 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:37.473912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:37.474716 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:37.474721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:37.474966 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:37.475730 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:49:37.475767 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:49:37.477671 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:49:37.479754 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:37.479762 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:37.479783 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:37.481115 (+ 1332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:37.481473 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:37.481476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:37.481479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:37.481734 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:37.482289 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:49:37.482299 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:49:37.482722 (+ 423us) client_negotiation.cc:769] Sending connection context
0810 01:49:37.482966 (+ 244us) client_negotiation.cc:240] Negotiation successful
0810 01:49:37.484445 (+ 1479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":182,"threads_started":1}
I20260810 01:49:37.487218 30374 heartbeater.cc:347] Connected to a master server at 127.9.77.190:33463
I20260810 01:49:37.487550 30374 heartbeater.cc:464] Registering TS with master...
I20260810 01:49:37.488379 30374 heartbeater.cc:511] Master 127.9.77.190:33463 requested a full tablet report, sending...
I20260810 01:49:37.490042 29644 ts_manager.cc:194] Registered new tserver with Master: 839eeb6e78994a20985097a9f62737c4 (127.9.77.131:41809)
I20260810 01:49:37.491446 29644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.131@KRBTEST.COM'} at 127.9.77.131:51697
Aug 10 01:49:37 dist-test-slave-kvfs krb5kdc[28046](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786326577, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:49:37.826082 29946 heartbeater.cc:464] Registering TS with master...
I20260810 01:49:37.826498 29946 heartbeater.cc:511] Master 127.9.77.190:33463 requested a full tablet report, sending...
I20260810 01:49:37.827702 29644 ts_manager.cc:194] Registered new tserver with Master: a0057c35515b48d596bcc6cb64f28713 (127.9.77.129:42371)
I20260810 01:49:37.828759 29644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.129@KRBTEST.COM'} at 127.9.77.129:43065
I20260810 01:49:38.156548 30143 heartbeater.cc:464] Registering TS with master...
I20260810 01:49:38.156988 30143 heartbeater.cc:511] Master 127.9.77.190:33463 requested a full tablet report, sending...
I20260810 01:49:38.157994 29644 ts_manager.cc:194] Registered new tserver with Master: d9c68f54081441939d921df17f8c285e (127.9.77.130:38413)
I20260810 01:49:38.158967 29644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.130@KRBTEST.COM'} at 127.9.77.130:43107
I20260810 01:49:38.165902 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 01:49:38.183013 30398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:38.168056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58824 (local address 127.9.77.190:33463)
0810 01:49:38.172267 (+ 4211us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:38.172272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:38.172291 (+ 19us) server_negotiation.cc:407] Connection header received
0810 01:49:38.172347 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:38.172350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:38.172426 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:38.172549 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:38.173610 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:38.174495 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:38.175765 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:38.176195 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:38.176316 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:38.178473 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:38.178478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:38.178481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:38.178511 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:38.180656 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:38.181214 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:38.181218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:38.181220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:38.181327 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:38.181678 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:38.181681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:38.181684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:38.181954 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:38.182102 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:38.182428 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 01:49:38.182585 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4104,"thread_start_us":98,"threads_started":1}
I20260810 01:49:38.188407 29644 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 01:49:38.194589 29644 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:49:38.219491 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:38.209835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:38413 (local address 127.0.0.1:54864)
0810 01:49:38.212880 (+ 3045us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:38.212963 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:38.213145 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:38.213887 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:38.213895 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:38.213927 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:49:38.214122 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:38.214131 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:38.215779 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:38.215783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:38.217314 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:38.217326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:38.218808 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:38.218811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:38.218935 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:38.218987 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 01:49:38.219115 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 01:49:38.219275 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2818,"thread_start_us":171,"threads_started":1}
I20260810 01:49:38.220336 30410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:38.210024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54864 (local address 127.9.77.130:38413)
0810 01:49:38.210459 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:38.210469 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:38.213116 (+ 2647us) server_negotiation.cc:407] Connection header received
0810 01:49:38.213358 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:38.213365 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:38.213668 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:38.213873 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:49:38.214283 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:38.215623 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:38.217490 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:38.218409 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:38.218500 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:38.218697 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:38.220001 (+ 1304us) server_negotiation.cc:299] Negotiation successful
0810 01:49:38.220169 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":146,"threads_started":1}
I20260810 01:49:38.225329 30017 tablet_service.cc:1467] Processing CreateTablet for tablet e4d4d10482974529978f07cd7dd317bb (DEFAULT_TABLE table=test-table [id=48c0c767afc042438f297b6c869e6229]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:49:38.227917 30017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4d4d10482974529978f07cd7dd317bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:49:38.229885 30417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:38.211330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:42371 (local address 127.0.0.1:59726)
0810 01:49:38.219738 (+ 8408us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:38.219757 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:38.219879 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:38.220918 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:38.220922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:38.220943 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:49:38.221067 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:38.221075 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:38.222612 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:38.222616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:38.223730 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:38.223742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:38.229351 (+ 5609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:38.229355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:38.229478 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:38.229510 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 01:49:38.229591 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 01:49:38.229697 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8327,"thread_start_us":115,"threads_started":1}
I20260810 01:49:38.231557 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:38.211584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59726 (local address 127.9.77.129:42371)
0810 01:49:38.217721 (+ 6137us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:38.217730 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:38.220007 (+ 2277us) server_negotiation.cc:407] Connection header received
0810 01:49:38.220154 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:38.220167 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:38.220499 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:38.220703 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:49:38.221228 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:38.222463 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:38.227975 (+ 5512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:38.228953 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:38.229081 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:38.229270 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:38.231220 (+ 1950us) server_negotiation.cc:299] Negotiation successful
0810 01:49:38.231382 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5979,"thread_start_us":79,"threads_started":1}
I20260810 01:49:38.235608 29823 tablet_service.cc:1467] Processing CreateTablet for tablet e4d4d10482974529978f07cd7dd317bb (DEFAULT_TABLE table=test-table [id=48c0c767afc042438f297b6c869e6229]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:49:38.237105 29823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4d4d10482974529978f07cd7dd317bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:49:38.246260 30407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:38.211864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:41809 (local address 127.0.0.1:43072)
0810 01:49:38.220910 (+ 9046us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:38.220925 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:38.222017 (+ 1092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:38.230984 (+ 8967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:38.230988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:38.231016 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:49:38.231133 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:38.231143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:38.243123 (+ 11980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:38.243128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:38.244424 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:38.244440 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:38.245750 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:38.245754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:38.245857 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:38.245898 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 01:49:38.245968 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 01:49:38.246074 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8972,"thread_start_us":115,"threads_started":1}
I20260810 01:49:38.247187 30418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:38.212197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43072 (local address 127.9.77.131:41809)
0810 01:49:38.230329 (+ 18132us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:38.230338 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:38.230371 (+ 33us) server_negotiation.cc:407] Connection header received
0810 01:49:38.230494 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:38.230504 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:38.230783 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:38.230975 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:49:38.241522 (+ 10547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:38.242938 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:38.244627 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:38.245580 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:38.246619 (+ 1039us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:38.246802 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:38.246865 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 01:49:38.247006 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17971,"thread_start_us":93,"threads_started":1}
I20260810 01:49:38.248673 30420 tablet_bootstrap.cc:492] T e4d4d10482974529978f07cd7dd317bb P a0057c35515b48d596bcc6cb64f28713: Bootstrap starting.
I20260810 01:49:38.251631 30420 tablet_bootstrap.cc:654] T e4d4d10482974529978f07cd7dd317bb P a0057c35515b48d596bcc6cb64f28713: Neither blocks nor log segments found. Creating new log.
I20260810 01:49:38.251235 30264 tablet_service.cc:1467] Processing CreateTablet for tablet e4d4d10482974529978f07cd7dd317bb (DEFAULT_TABLE table=test-table [id=48c0c767afc042438f297b6c869e6229]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:49:38.252703 30264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4d4d10482974529978f07cd7dd317bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:49:38.252739 30420 log.cc:824] T e4d4d10482974529978f07cd7dd317bb P a0057c35515b48d596bcc6cb64f28713: Log is configured to *not* fsync() on all Append() calls
I20260810 01:49:38.253398 30419 tablet_bootstrap.cc:492] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e: Bootstrap starting.
I20260810 01:49:38.255330 30420 tablet_bootstrap.cc:492] T e4d4d10482974529978f07cd7dd317bb P a0057c35515b48d596bcc6cb64f28713: No bootstrap required, opened a new log
I20260810 01:49:38.255579 30420 ts_tablet_manager.cc:1397] T e4d4d10482974529978f07cd7dd317bb P a0057c35515b48d596bcc6cb64f28713: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 01:49:38.260385 30420 raft_consensus.cc:348] T e4d4d10482974529978f07cd7dd317bb P a0057c35515b48d596bcc6cb64f28713 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9c68f54081441939d921df17f8c285e" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38413 } } peers { permanent_uuid: "a0057c35515b48d596bcc6cb64f28713" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 42371 } } peers { permanent_uuid: "839eeb6e78994a20985097a9f62737c4" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 41809 } }
I20260810 01:49:38.260741 30420 raft_consensus.cc:374] T e4d4d10482974529978f07cd7dd317bb P a0057c35515b48d596bcc6cb64f28713 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:49:38.260804 30420 raft_consensus.cc:729] T e4d4d10482974529978f07cd7dd317bb P a0057c35515b48d596bcc6cb64f28713 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0057c35515b48d596bcc6cb64f28713, State: Initialized, Role: FOLLOWER
I20260810 01:49:38.261433 30420 consensus_queue.cc:260] T e4d4d10482974529978f07cd7dd317bb P a0057c35515b48d596bcc6cb64f28713 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9c68f54081441939d921df17f8c285e" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38413 } } peers { permanent_uuid: "a0057c35515b48d596bcc6cb64f28713" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 42371 } } peers { permanent_uuid: "839eeb6e78994a20985097a9f62737c4" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 41809 } }
I20260810 01:49:38.262763 30420 ts_tablet_manager.cc:1428] T e4d4d10482974529978f07cd7dd317bb P a0057c35515b48d596bcc6cb64f28713: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 01:49:38.262776 30419 tablet_bootstrap.cc:654] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e: Neither blocks nor log segments found. Creating new log.
I20260810 01:49:38.263252 29946 heartbeater.cc:503] Master 127.9.77.190:33463 was elected leader, sending a full tablet report...
I20260810 01:49:38.264046 30419 log.cc:824] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e: Log is configured to *not* fsync() on all Append() calls
I20260810 01:49:38.265326 30426 tablet_bootstrap.cc:492] T e4d4d10482974529978f07cd7dd317bb P 839eeb6e78994a20985097a9f62737c4: Bootstrap starting.
I20260810 01:49:38.270500 30426 tablet_bootstrap.cc:654] T e4d4d10482974529978f07cd7dd317bb P 839eeb6e78994a20985097a9f62737c4: Neither blocks nor log segments found. Creating new log.
I20260810 01:49:38.271131 30419 tablet_bootstrap.cc:492] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e: No bootstrap required, opened a new log
I20260810 01:49:38.271405 30419 ts_tablet_manager.cc:1397] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20260810 01:49:38.271545 30426 log.cc:824] T e4d4d10482974529978f07cd7dd317bb P 839eeb6e78994a20985097a9f62737c4: Log is configured to *not* fsync() on all Append() calls
I20260810 01:49:38.275663 30419 raft_consensus.cc:348] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9c68f54081441939d921df17f8c285e" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38413 } } peers { permanent_uuid: "a0057c35515b48d596bcc6cb64f28713" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 42371 } } peers { permanent_uuid: "839eeb6e78994a20985097a9f62737c4" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 41809 } }
I20260810 01:49:38.275952 30419 raft_consensus.cc:374] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:49:38.276057 30419 raft_consensus.cc:729] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9c68f54081441939d921df17f8c285e, State: Initialized, Role: FOLLOWER
I20260810 01:49:38.276691 30419 consensus_queue.cc:260] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9c68f54081441939d921df17f8c285e" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38413 } } peers { permanent_uuid: "a0057c35515b48d596bcc6cb64f28713" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 42371 } } peers { permanent_uuid: "839eeb6e78994a20985097a9f62737c4" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 41809 } }
I20260810 01:49:38.278105 30419 ts_tablet_manager.cc:1428] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260810 01:49:38.278573 30143 heartbeater.cc:503] Master 127.9.77.190:33463 was elected leader, sending a full tablet report...
W20260810 01:49:38.286029 29947 tablet.cc:2369] T e4d4d10482974529978f07cd7dd317bb P a0057c35515b48d596bcc6cb64f28713: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 01:49:38.286371 30426 tablet_bootstrap.cc:492] T e4d4d10482974529978f07cd7dd317bb P 839eeb6e78994a20985097a9f62737c4: No bootstrap required, opened a new log
I20260810 01:49:38.286674 30426 ts_tablet_manager.cc:1397] T e4d4d10482974529978f07cd7dd317bb P 839eeb6e78994a20985097a9f62737c4: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.002s
I20260810 01:49:38.292315 30426 raft_consensus.cc:348] T e4d4d10482974529978f07cd7dd317bb P 839eeb6e78994a20985097a9f62737c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9c68f54081441939d921df17f8c285e" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38413 } } peers { permanent_uuid: "a0057c35515b48d596bcc6cb64f28713" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 42371 } } peers { permanent_uuid: "839eeb6e78994a20985097a9f62737c4" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 41809 } }
I20260810 01:49:38.292721 30426 raft_consensus.cc:374] T e4d4d10482974529978f07cd7dd317bb P 839eeb6e78994a20985097a9f62737c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:49:38.292793 30426 raft_consensus.cc:729] T e4d4d10482974529978f07cd7dd317bb P 839eeb6e78994a20985097a9f62737c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 839eeb6e78994a20985097a9f62737c4, State: Initialized, Role: FOLLOWER
I20260810 01:49:38.293489 30426 consensus_queue.cc:260] T e4d4d10482974529978f07cd7dd317bb P 839eeb6e78994a20985097a9f62737c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9c68f54081441939d921df17f8c285e" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38413 } } peers { permanent_uuid: "a0057c35515b48d596bcc6cb64f28713" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 42371 } } peers { permanent_uuid: "839eeb6e78994a20985097a9f62737c4" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 41809 } }
I20260810 01:49:38.294956 30426 ts_tablet_manager.cc:1428] T e4d4d10482974529978f07cd7dd317bb P 839eeb6e78994a20985097a9f62737c4: Time spent starting tablet: real 0.008s user 0.008s sys 0.001s
I20260810 01:49:38.295459 30374 heartbeater.cc:503] Master 127.9.77.190:33463 was elected leader, sending a full tablet report...
Aug 10 01:49:38 dist-test-slave-kvfs krb5kdc[28046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326577, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for HTTP/127.9.77.148@KRBTEST.COM
W20260810 01:49:38.381290 30152 tablet.cc:2369] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 01:49:38.466434 30375 tablet.cc:2369] T e4d4d10482974529978f07cd7dd317bb P 839eeb6e78994a20985097a9f62737c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 01:49:38.469269 30430 raft_consensus.cc:482] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:49:38.469578 30430 raft_consensus.cc:504] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9c68f54081441939d921df17f8c285e" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38413 } } peers { permanent_uuid: "a0057c35515b48d596bcc6cb64f28713" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 42371 } } peers { permanent_uuid: "839eeb6e78994a20985097a9f62737c4" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 41809 } }
I20260810 01:49:38.471381 30430 leader_election.cc:288] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0057c35515b48d596bcc6cb64f28713 (127.9.77.129:42371), 839eeb6e78994a20985097a9f62737c4 (127.9.77.131:41809)
I20260810 01:49:38.480751 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:38.473247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:59881 (local address 127.9.77.129:42371)
0810 01:49:38.473385 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:38.473391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:38.475515 (+ 2124us) server_negotiation.cc:407] Connection header received
0810 01:49:38.475783 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:38.475786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:38.475852 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:38.475905 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:49:38.476501 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:38.477311 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:38.479147 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:38.479818 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:38.479951 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:38.480047 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:38.480464 (+ 417us) server_negotiation.cc:299] Negotiation successful
0810 01:49:38.480573 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 01:49:38.481354 30432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:38.471945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:42371 (local address 127.9.77.130:59881)
0810 01:49:38.475376 (+ 3431us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:38.475396 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:38.475632 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:38.476120 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:38.476124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:38.476144 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:49:38.476347 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:38.476356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:38.477506 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:38.477510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:38.478989 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:38.479002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:38.480165 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:38.480168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:38.480290 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:38.480331 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 01:49:38.481072 (+ 741us) client_negotiation.cc:240] Negotiation successful
0810 01:49:38.481173 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3323,"thread_start_us":198,"threads_started":1}
I20260810 01:49:38.482426 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d4d10482974529978f07cd7dd317bb" candidate_uuid: "d9c68f54081441939d921df17f8c285e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0057c35515b48d596bcc6cb64f28713" is_pre_election: true
I20260810 01:49:38.482896 29875 raft_consensus.cc:2454] T e4d4d10482974529978f07cd7dd317bb P a0057c35515b48d596bcc6cb64f28713 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9c68f54081441939d921df17f8c285e in term 0.
I20260810 01:49:38.483568 29983 leader_election.cc:302] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0057c35515b48d596bcc6cb64f28713, d9c68f54081441939d921df17f8c285e; no voters:
I20260810 01:49:38.484009 30430 raft_consensus.cc:2781] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:49:38.484180 30430 raft_consensus.cc:482] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:49:38.484232 30430 raft_consensus.cc:3037] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:49:38.485903 30430 raft_consensus.cc:504] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9c68f54081441939d921df17f8c285e" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38413 } } peers { permanent_uuid: "a0057c35515b48d596bcc6cb64f28713" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 42371 } } peers { permanent_uuid: "839eeb6e78994a20985097a9f62737c4" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 41809 } }
I20260810 01:49:38.486413 30430 leader_election.cc:288] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e [CANDIDATE]: Term 1 election: Requested vote from peers a0057c35515b48d596bcc6cb64f28713 (127.9.77.129:42371), 839eeb6e78994a20985097a9f62737c4 (127.9.77.131:41809)
I20260810 01:49:38.487051 29875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d4d10482974529978f07cd7dd317bb" candidate_uuid: "d9c68f54081441939d921df17f8c285e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0057c35515b48d596bcc6cb64f28713"
I20260810 01:49:38.487215 29875 raft_consensus.cc:3037] T e4d4d10482974529978f07cd7dd317bb P a0057c35515b48d596bcc6cb64f28713 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:49:38.488903 29875 raft_consensus.cc:2454] T e4d4d10482974529978f07cd7dd317bb P a0057c35515b48d596bcc6cb64f28713 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9c68f54081441939d921df17f8c285e in term 1.
I20260810 01:49:38.489311 29983 leader_election.cc:302] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0057c35515b48d596bcc6cb64f28713, d9c68f54081441939d921df17f8c285e; no voters:
I20260810 01:49:38.489518 30430 raft_consensus.cc:2781] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:49:38.489794 30430 raft_consensus.cc:686] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e [term 1 LEADER]: Becoming Leader. State: Replica: d9c68f54081441939d921df17f8c285e, State: Running, Role: LEADER
I20260810 01:49:38.490304 30430 consensus_queue.cc:237] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9c68f54081441939d921df17f8c285e" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38413 } } peers { permanent_uuid: "a0057c35515b48d596bcc6cb64f28713" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 42371 } } peers { permanent_uuid: "839eeb6e78994a20985097a9f62737c4" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 41809 } }
I20260810 01:49:38.494908 29643 catalog_manager.cc:5495] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e reported cstate change: term changed from 0 to 1, leader changed from <none> to d9c68f54081441939d921df17f8c285e (127.9.77.130). New cstate: current_term: 1 leader_uuid: "d9c68f54081441939d921df17f8c285e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9c68f54081441939d921df17f8c285e" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0057c35515b48d596bcc6cb64f28713" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 42371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "839eeb6e78994a20985097a9f62737c4" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 41809 } health_report { overall_health: UNKNOWN } } }
I20260810 01:49:38.496670 30418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:38.472691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:49705 (local address 127.9.77.131:41809)
0810 01:49:38.473031 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:38.473036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:38.481717 (+ 8681us) server_negotiation.cc:407] Connection header received
0810 01:49:38.481983 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:38.481987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:38.482066 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:38.482127 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:49:38.482588 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:38.483416 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:38.495034 (+ 11618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:38.495709 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:38.495837 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:38.495907 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:38.496382 (+ 475us) server_negotiation.cc:299] Negotiation successful
0810 01:49:38.496507 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 01:49:38.497202 30434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:38.472570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:41809 (local address 127.9.77.130:49705)
0810 01:49:38.481374 (+ 8804us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:38.481394 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:38.481809 (+ 415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:38.482301 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:38.482304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:38.482324 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:49:38.482442 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:38.482448 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:38.483787 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:38.483791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:38.494783 (+ 10992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:38.494799 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:38.496066 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:38.496070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:38.496183 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:38.496211 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 01:49:38.496919 (+ 708us) client_negotiation.cc:240] Negotiation successful
0810 01:49:38.497034 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8723,"thread_start_us":140,"threads_started":1}
I20260810 01:49:38.498144 30296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d4d10482974529978f07cd7dd317bb" candidate_uuid: "d9c68f54081441939d921df17f8c285e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "839eeb6e78994a20985097a9f62737c4" is_pre_election: true
I20260810 01:49:38.498581 30296 raft_consensus.cc:2454] T e4d4d10482974529978f07cd7dd317bb P 839eeb6e78994a20985097a9f62737c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9c68f54081441939d921df17f8c285e in term 0.
I20260810 01:49:38.499336 30292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d4d10482974529978f07cd7dd317bb" candidate_uuid: "d9c68f54081441939d921df17f8c285e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "839eeb6e78994a20985097a9f62737c4"
I20260810 01:49:38.499467 30292 raft_consensus.cc:3037] T e4d4d10482974529978f07cd7dd317bb P 839eeb6e78994a20985097a9f62737c4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:49:38.501268 30292 raft_consensus.cc:2454] T e4d4d10482974529978f07cd7dd317bb P 839eeb6e78994a20985097a9f62737c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9c68f54081441939d921df17f8c285e in term 1.
I20260810 01:49:38.538493 30410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:38.531473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54876 (local address 127.9.77.130:38413)
0810 01:49:38.531663 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:38.531668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:38.531686 (+ 18us) server_negotiation.cc:407] Connection header received
0810 01:49:38.531809 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:38.531812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:38.531898 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:38.532645 (+ 747us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:49:38.532800 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:38.533677 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:38.535245 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:38.535598 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:38.535710 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:38.536291 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:49:38.537139 (+ 848us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:49:38.537282 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:38.538174 (+ 892us) server_negotiation.cc:299] Negotiation successful
0810 01:49:38.538269 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260810 01:49:38.549316 30292 raft_consensus.cc:1261] T e4d4d10482974529978f07cd7dd317bb P 839eeb6e78994a20985097a9f62737c4 [term 1 FOLLOWER]: Refusing update from remote peer d9c68f54081441939d921df17f8c285e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:49:38.549336 29875 raft_consensus.cc:1261] T e4d4d10482974529978f07cd7dd317bb P a0057c35515b48d596bcc6cb64f28713 [term 1 FOLLOWER]: Refusing update from remote peer d9c68f54081441939d921df17f8c285e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:49:38.550331 30435 consensus_queue.cc:1035] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0057c35515b48d596bcc6cb64f28713" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 42371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:49:38.550592 30430 consensus_queue.cc:1035] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e [LEADER]: Connected to new peer: Peer: permanent_uuid: "839eeb6e78994a20985097a9f62737c4" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 41809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:49:38.566823 30438 mvcc.cc:205] Tried to move back new op lower bound from 7316793665724432384 to 7316793665500577792. Current Snapshot: MvccSnapshot[applied={T|T < 7316793665724432384}]
I20260810 01:49:38.567359 30439 mvcc.cc:205] Tried to move back new op lower bound from 7316793665724432384 to 7316793665500577792. Current Snapshot: MvccSnapshot[applied={T|T < 7316793665724432384}]
I20260810 01:49:38.571597 30441 mvcc.cc:205] Tried to move back new op lower bound from 7316793665724432384 to 7316793665500577792. Current Snapshot: MvccSnapshot[applied={T|T < 7316793665724432384}]
I20260810 01:49:38.584708 29644 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:49:38.584985 29644 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:49:38.588807 29644 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3bac3a48e8874ba88778744fe9eaa9fc: Sending DeleteTablet for 3 replicas of tablet e4d4d10482974529978f07cd7dd317bb
I20260810 01:49:38.590545 29823 tablet_service.cc:1514] Processing DeleteTablet for tablet e4d4d10482974529978f07cd7dd317bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:49:38 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:59726
I20260810 01:49:38.590545 30017 tablet_service.cc:1514] Processing DeleteTablet for tablet e4d4d10482974529978f07cd7dd317bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:49:38 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:54864
I20260810 01:49:38.591886 30264 tablet_service.cc:1514] Processing DeleteTablet for tablet e4d4d10482974529978f07cd7dd317bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:49:38 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:43072
I20260810 01:49:38.595333 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 29709
I20260810 01:49:38.604404 30455 tablet_replica.cc:330] T e4d4d10482974529978f07cd7dd317bb P 839eeb6e78994a20985097a9f62737c4: stopping tablet replica
I20260810 01:49:38.605010 30455 raft_consensus.cc:2229] T e4d4d10482974529978f07cd7dd317bb P 839eeb6e78994a20985097a9f62737c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:49:38.605537 30455 raft_consensus.cc:2258] T e4d4d10482974529978f07cd7dd317bb P 839eeb6e78994a20985097a9f62737c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:49:38.608596 30454 tablet_replica.cc:330] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e: stopping tablet replica
I20260810 01:49:38.609153 30454 raft_consensus.cc:2229] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e [term 1 LEADER]: Raft consensus shutting down.
I20260810 01:49:38.609760 30454 raft_consensus.cc:2258] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:49:38.612104 30454 ts_tablet_manager.cc:1905] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:49:38.614058 30455 ts_tablet_manager.cc:1905] T e4d4d10482974529978f07cd7dd317bb P 839eeb6e78994a20985097a9f62737c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:49:38.619063 30454 ts_tablet_manager.cc:1918] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:49:38.619201 30454 log.cc:1196] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-1/wal/wals/e4d4d10482974529978f07cd7dd317bb
I20260810 01:49:38.619581 30454 ts_tablet_manager.cc:1939] T e4d4d10482974529978f07cd7dd317bb P d9c68f54081441939d921df17f8c285e: Deleting consensus metadata
I20260810 01:49:38.622161 30455 ts_tablet_manager.cc:1918] T e4d4d10482974529978f07cd7dd317bb P 839eeb6e78994a20985097a9f62737c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:49:38.622298 30455 log.cc:1196] T e4d4d10482974529978f07cd7dd317bb P 839eeb6e78994a20985097a9f62737c4: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786326392877349-9526-0/minicluster-data/ts-2/wal/wals/e4d4d10482974529978f07cd7dd317bb
I20260810 01:49:38.622664 30455 ts_tablet_manager.cc:1939] T e4d4d10482974529978f07cd7dd317bb P 839eeb6e78994a20985097a9f62737c4: Deleting consensus metadata
I20260810 01:49:38.625238 29619 catalog_manager.cc:4841] TS d9c68f54081441939d921df17f8c285e (127.9.77.130:38413): tablet e4d4d10482974529978f07cd7dd317bb (table test-table [id=48c0c767afc042438f297b6c869e6229]) successfully deleted
W20260810 01:49:38.625840 29618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.77.129:42371 (error 108)
I20260810 01:49:38.626544 29616 catalog_manager.cc:4841] TS 839eeb6e78994a20985097a9f62737c4 (127.9.77.131:41809): tablet e4d4d10482974529978f07cd7dd317bb (table test-table [id=48c0c767afc042438f297b6c869e6229]) successfully deleted
I20260810 01:49:38.628242 30407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:38.627835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:42371 (local address 127.0.0.1:59738)
0810 01:49:38.628058 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:38.628135 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.129:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260810 01:49:38.628545 29618 catalog_manager.cc:4584] TS a0057c35515b48d596bcc6cb64f28713 (127.9.77.129:42371): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e4d4d10482974529978f07cd7dd317bb: Network error: Client connection negotiation failed: client connection to 127.9.77.129:42371: connect: Connection refused (error 111)
I20260810 01:49:38.629014 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 29950
I20260810 01:49:38.651374 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 30153
I20260810 01:49:38.675031 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 29587
I20260810 01:49:38.691339 9526 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 01:49:39.038198 9526 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 01:49:39.038354 9526 mini_ranger.cc:66] Stopping Ranger...
I20260810 01:49:39.388337 9526 mini_ranger.cc:68] Stopped Ranger
2026-08-10 01:49:39.388 UTC [28091] LOG: received smart shutdown request
2026-08-10 01:49:39.393 UTC [28091] LOG: background worker "logical replication launcher" (PID 28109) exited with exit code 1
2026-08-10 01:49:39.393 UTC [28097] LOG: shutting down
2026-08-10 01:49:39.583 UTC [28091] LOG: database system is shut down
2026-08-10T01:49:39Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55688 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:49:39 dist-test-slave-kvfs krb5kdc[30471](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:49:39 dist-test-slave-kvfs krb5kdc[30471](info): set up 2 sockets
Aug 10 01:49:39 dist-test-slave-kvfs krb5kdc[30471](info): commencing operation
krb5kdc: starting...
W20260810 01:49:41.009358 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.267s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:49:41 dist-test-slave-kvfs krb5kdc[30471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:49:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:49:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:49:41.156203 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:45681
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:37017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:45681,127.9.77.189:40369,127.9.77.188:39015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:49:41.297215 30491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:49:41.297586 30491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:49:41.297633 30491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:49:41.301915 30491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:49:41.302002 30491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:49:41.302022 30491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:49:41.302042 30491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:49:41.302057 30491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:49:41.307894 30491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:37017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:45681,127.9.77.189:40369,127.9.77.188:39015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:45681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:49:41.309481 30491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:49:41.310883 30491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 01:49:41.321359 30491 server_base.cc:1034] running on GCE node
W20260810 01:49:41.321302 30501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:41.321301 30498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:41.321301 30497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:41.322434 30491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:49:41.323889 30491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:49:41.325063 30491 hybrid_clock.cc:648] HybridClock initialized: now 1786326581325034 us; error 46 us; skew 500 ppm
Aug 10 01:49:41 dist-test-slave-kvfs krb5kdc[30471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326581, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:49:41.328434 30491 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:49:41.329592 30491 webserver.cc:460] Webserver started at http://127.9.77.190:43785/ using document root <none> and password file <none>
I20260810 01:49:41.330328 30491 fs_manager.cc:359] Metadata directory not provided
I20260810 01:49:41.330401 30491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:49:41.330687 30491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:49:41.333441 30491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "4bf9a150e9ef4db5ab28be9ede40fe90"
format_stamp: "Formatted at 2026-08-10 01:49:41 on dist-test-slave-kvfs"
server_key: "c72b59f7765086f1d86b617b1b64629c"
server_key_iv: "3e5f318f97154a2efc95f16c047a4709"
server_key_version: "encryptionkey@0"
I20260810 01:49:41.334259 30491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "4bf9a150e9ef4db5ab28be9ede40fe90"
format_stamp: "Formatted at 2026-08-10 01:49:41 on dist-test-slave-kvfs"
server_key: "c72b59f7765086f1d86b617b1b64629c"
server_key_iv: "3e5f318f97154a2efc95f16c047a4709"
server_key_version: "encryptionkey@0"
I20260810 01:49:41.339214 30491 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 01:49:41.342474 30513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:41.344121 30491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 01:49:41.344285 30491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "4bf9a150e9ef4db5ab28be9ede40fe90"
format_stamp: "Formatted at 2026-08-10 01:49:41 on dist-test-slave-kvfs"
server_key: "c72b59f7765086f1d86b617b1b64629c"
server_key_iv: "3e5f318f97154a2efc95f16c047a4709"
server_key_version: "encryptionkey@0"
I20260810 01:49:41.344426 30491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:49:41.370857 30491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:49:41.375059 30491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:49:41.375334 30491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:49:41.385146 30491 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:45681
I20260810 01:49:41.385180 30599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:45681 every 8 connection(s)
I20260810 01:49:41.386385 30491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:49:41.390136 30600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:49:41.395620 30600 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45681 } has no permanent_uuid. Determining permanent_uuid...
I20260810 01:49:41.396349 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 30491
I20260810 01:49:41.396526 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:49:41.396875 9526 external_mini_cluster.cc:1428] Setting key ed0173dd5c7aacdbf2414b51314e48b6
Aug 10 01:49:41 dist-test-slave-kvfs krb5kdc[30471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326581, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.77.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.189@KRBTEST.COM" created.
I20260810 01:49:41.415074 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.398925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45681 (local address 127.0.0.1:50300)
0810 01:49:41.399691 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.399779 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:41.400893 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:41.401711 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:41.401717 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:41.402267 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:49:41.402569 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:41.402585 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.403871 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.403874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:41.404755 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:41.404766 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.405484 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.405488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:41.405743 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:41.406539 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:49:41.406559 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:49:41.408683 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:49:41.411091 (+ 2408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:41.411101 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:41.411112 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:41.411493 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:41.411912 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:41.411915 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:41.411917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:41.412133 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:41.413030 (+ 897us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:49:41.413038 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:49:41.413252 (+ 214us) client_negotiation.cc:769] Sending connection context
0810 01:49:41.413486 (+ 234us) client_negotiation.cc:240] Negotiation successful
0810 01:49:41.414009 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":137,"thread_start_us":187,"threads_started":1}
I20260810 01:49:41.415072 30606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.398925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50300 (local address 127.9.77.190:45681)
0810 01:49:41.399647 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:41.399668 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:41.400014 (+ 346us) server_negotiation.cc:407] Connection header received
0810 01:49:41.401262 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:41.401267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:41.401487 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:41.401720 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:41.402746 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.403703 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.404894 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.405305 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.405499 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:41.408969 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:41.408977 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:41.408991 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:41.409017 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:41.410959 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:41.411643 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:41.411649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:41.411650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:41.411794 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:41.412284 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:41.412287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:41.412288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:41.412889 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:41.413019 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:41.413647 (+ 628us) server_negotiation.cc:299] Negotiation successful
0810 01:49:41.413889 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":577,"spinlock_wait_cycles":46976,"thread_start_us":203,"threads_started":1}
I20260810 01:49:41.418283 30600 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40369 } has no permanent_uuid. Determining permanent_uuid...
I20260810 01:49:41.419238 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.418923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:40369 (local address 127.0.0.1:43648)
0810 01:49:41.419067 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.419155 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.189:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 01:49:41.419450 30516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.77.189:40369: connect: Connection refused (error 111)
I20260810 01:49:41.420673 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.420432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:40369 (local address 127.0.0.1:43662)
0810 01:49:41.420555 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.420611 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.189:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 01:49:41.420828 30600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.189:40369: Network error: Client connection negotiation failed: client connection to 127.9.77.189:40369: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.189.keytab.
Entry for principal kudu/127.9.77.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.189.keytab.
I20260810 01:49:41.435279 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.189:40369
--webserver_interface=127.9.77.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.189.keytab
--principal=kudu/127.9.77.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:37017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:45681,127.9.77.189:40369,127.9.77.188:39015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 01:49:41.470155 30600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40369 } attempt: 1
I20260810 01:49:41.471575 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.471118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:40369 (local address 127.0.0.1:43664)
0810 01:49:41.471358 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.471449 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.189:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260810 01:49:41.472641 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.472309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:40369 (local address 127.0.0.1:43680)
0810 01:49:41.472464 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.472534 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.189:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 01:49:41.472869 30600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.189:40369: Network error: Client connection negotiation failed: client connection to 127.9.77.189:40369: connect: Connection refused (error 111)
I20260810 01:49:41.541107 30600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40369 } attempt: 2
I20260810 01:49:41.542424 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.541942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:40369 (local address 127.0.0.1:43694)
0810 01:49:41.542180 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.542272 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.189:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260810 01:49:41.543367 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.543082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:40369 (local address 127.0.0.1:43698)
0810 01:49:41.543227 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.543301 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.189:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260810 01:49:41.543581 30600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.189:40369: Network error: Client connection negotiation failed: client connection to 127.9.77.189:40369: connect: Connection refused (error 111)
W20260810 01:49:41.579864 30610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:49:41.580309 30610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:49:41.580356 30610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:49:41.584924 30610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:49:41.585011 30610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:49:41.585032 30610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:49:41.585050 30610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:49:41.585067 30610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:49:41.591248 30610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:37017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:45681,127.9.77.189:40369,127.9.77.188:39015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.189.keytab
--principal=kudu/127.9.77.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.189:40369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.77.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:49:41.592820 30610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:49:41.594157 30610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:49:41.604560 30620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:41.604492 30619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:41.604492 30622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:41.604710 30610 server_base.cc:1034] running on GCE node
I20260810 01:49:41.605633 30610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:49:41.607105 30610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:49:41.608323 30610 hybrid_clock.cc:648] HybridClock initialized: now 1786326581608292 us; error 46 us; skew 500 ppm
Aug 10 01:49:41 dist-test-slave-kvfs krb5kdc[30471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326581, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:49:41.611601 30610 init.cc:377] Logged in from keytab as kudu/127.9.77.189@KRBTEST.COM (short username kudu)
I20260810 01:49:41.612771 30610 webserver.cc:460] Webserver started at http://127.9.77.189:35955/ using document root <none> and password file <none>
I20260810 01:49:41.613471 30610 fs_manager.cc:359] Metadata directory not provided
I20260810 01:49:41.613574 30610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:49:41.613859 30610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:49:41.616456 30610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-1/data/instance:
uuid: "6b992eba730945beb0fb5ee818bf73d6"
format_stamp: "Formatted at 2026-08-10 01:49:41 on dist-test-slave-kvfs"
server_key: "d6b56820749ca6bd6b9feb1d5c8c3285"
server_key_iv: "e2284e99ffec311d288aa5d74b3bf8b4"
server_key_version: "encryptionkey@0"
I20260810 01:49:41.617120 30610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-1/wal/instance:
uuid: "6b992eba730945beb0fb5ee818bf73d6"
format_stamp: "Formatted at 2026-08-10 01:49:41 on dist-test-slave-kvfs"
server_key: "d6b56820749ca6bd6b9feb1d5c8c3285"
server_key_iv: "e2284e99ffec311d288aa5d74b3bf8b4"
server_key_version: "encryptionkey@0"
I20260810 01:49:41.621594 30610 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 01:49:41.624579 30631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:41.626129 30610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:49:41.626282 30610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-1/wal
uuid: "6b992eba730945beb0fb5ee818bf73d6"
format_stamp: "Formatted at 2026-08-10 01:49:41 on dist-test-slave-kvfs"
server_key: "d6b56820749ca6bd6b9feb1d5c8c3285"
server_key_iv: "e2284e99ffec311d288aa5d74b3bf8b4"
server_key_version: "encryptionkey@0"
I20260810 01:49:41.626425 30610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:49:41.634778 30600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40369 } attempt: 3
I20260810 01:49:41.635797 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.635468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:40369 (local address 127.0.0.1:43700)
0810 01:49:41.635629 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.635696 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.189:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 01:49:41.636606 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.636356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:40369 (local address 127.0.0.1:43704)
0810 01:49:41.636477 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.636537 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.189:40369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 01:49:41.636744 30600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.189:40369: Network error: Client connection negotiation failed: client connection to 127.9.77.189:40369: connect: Connection refused (error 111)
I20260810 01:49:41.642879 30610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:49:41.646425 30610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:49:41.646646 30610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:49:41.655517 30610 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.189:40369
I20260810 01:49:41.655558 30716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.189:40369 every 8 connection(s)
I20260810 01:49:41.656781 30610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-1/data/info.pb
I20260810 01:49:41.660372 30718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:49:41.665042 30718 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45681 } has no permanent_uuid. Determining permanent_uuid...
I20260810 01:49:41.665761 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 30610
I20260810 01:49:41.665938 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-1/wal/instance
I20260810 01:49:41.666283 9526 external_mini_cluster.cc:1428] Setting key fc9f420a5eb68c9741b5c13776a618af
Aug 10 01:49:41 dist-test-slave-kvfs krb5kdc[30471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326581, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.189@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.77.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.188@KRBTEST.COM" created.
I20260810 01:49:41.682374 30606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.667606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50308 (local address 127.9.77.190:45681)
0810 01:49:41.667834 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:41.667841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:41.668944 (+ 1103us) server_negotiation.cc:407] Connection header received
0810 01:49:41.670088 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:41.670094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:41.670177 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:41.670286 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:41.671694 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.672640 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.674060 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.674541 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.674682 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:41.677646 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:41.677651 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:41.677653 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:41.677685 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:41.679510 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:41.680315 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:41.680320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:41.680323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:41.680456 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:41.680910 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:41.680914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:41.680915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:41.681152 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:41.681279 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:41.681966 (+ 687us) server_negotiation.cc:299] Negotiation successful
0810 01:49:41.682131 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 01:49:41.683663 30724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.667995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45681 (local address 127.0.0.1:50308)
0810 01:49:41.668707 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.668786 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:41.669840 (+ 1054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:41.670488 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:41.670495 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:41.670928 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:49:41.671520 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:41.671536 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.672795 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.672798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:41.673913 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:41.673924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.674742 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.674745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:41.674938 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:41.675688 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:49:41.675708 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:49:41.677433 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:49:41.679673 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:41.679681 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:41.679709 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:41.680183 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:41.680582 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:41.680585 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:41.680588 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:41.680775 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:41.681284 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:49:41.681295 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:49:41.681651 (+ 356us) client_negotiation.cc:769] Sending connection context
0810 01:49:41.681885 (+ 234us) client_negotiation.cc:240] Negotiation successful
0810 01:49:41.682335 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":140,"threads_started":1}
I20260810 01:49:41.685720 30718 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40369 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 01:49:41 dist-test-slave-kvfs krb5kdc[30471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326581, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.189@KRBTEST.COM for kudu/127.9.77.189@KRBTEST.COM
I20260810 01:49:41.696456 30724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.686494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:40369 (local address 127.0.0.1:43708)
0810 01:49:41.686634 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.686656 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:41.686727 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:41.687426 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:41.687429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:41.687645 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:49:41.687729 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:41.687734 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.688853 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.688859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:41.689438 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:41.689446 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.690081 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.690084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:41.690290 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:41.690897 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:49:41.690909 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:49:41.692515 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:49:41.694776 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:41.694783 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:41.694788 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:41.695058 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:41.695375 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:41.695377 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:41.695379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:41.695461 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:41.695967 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:49:41.695969 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:49:41.696102 (+ 133us) client_negotiation.cc:769] Sending connection context
0810 01:49:41.696188 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 01:49:41.696304 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":68}
I20260810 01:49:41.696802 30728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.686620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43708 (local address 127.9.77.189:40369)
0810 01:49:41.687046 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:41.687056 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:41.687076 (+ 20us) server_negotiation.cc:407] Connection header received
0810 01:49:41.687114 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:41.687118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:41.687298 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:41.687473 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:41.687843 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.688718 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.689557 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.689962 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.690084 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:41.692685 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:41.692695 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:41.692710 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:41.692732 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:41.694657 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:41.695165 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:41.695170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:41.695171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:41.695254 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:41.695560 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:41.695563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:41.695564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:41.695859 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:41.695991 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:41.696335 (+ 344us) server_negotiation.cc:299] Negotiation successful
0810 01:49:41.696478 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":331,"thread_start_us":216,"threads_started":1}
I20260810 01:49:41.698966 30718 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } has no permanent_uuid. Determining permanent_uuid...
I20260810 01:49:41.699767 30724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.699494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:39015 (local address 127.0.0.1:36168)
0810 01:49:41.699621 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.699708 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:39015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 01:49:41.699966 30634 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.77.188:39015: connect: Connection refused (error 111)
I20260810 01:49:41.700977 30724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.700756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:39015 (local address 127.0.0.1:36176)
0810 01:49:41.700869 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.700926 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:39015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 01:49:41.701150 30718 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.188:39015: Network error: Client connection negotiation failed: client connection to 127.9.77.188:39015: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.188.keytab.
Entry for principal kudu/127.9.77.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.188.keytab.
I20260810 01:49:41.703321 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.188:39015
--webserver_interface=127.9.77.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.188.keytab
--principal=kudu/127.9.77.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:37017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:45681,127.9.77.189:40369,127.9.77.188:39015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 01:49:41.750422 30718 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } attempt: 1
I20260810 01:49:41.751652 30724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.751238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:39015 (local address 127.0.0.1:36192)
0810 01:49:41.751444 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.751538 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:39015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 01:49:41.752547 30724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.752289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:39015 (local address 127.0.0.1:36206)
0810 01:49:41.752427 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.752483 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:39015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 01:49:41.752717 30718 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.188:39015: Network error: Client connection negotiation failed: client connection to 127.9.77.188:39015: connect: Connection refused (error 111)
I20260810 01:49:41.779944 30600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40369 } attempt: 4
Aug 10 01:49:41 dist-test-slave-kvfs krb5kdc[30471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326581, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for kudu/127.9.77.189@KRBTEST.COM
I20260810 01:49:41.791721 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.780869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:40369 (local address 127.0.0.1:43718)
0810 01:49:41.781154 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.781174 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:41.781339 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:41.781673 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:41.781677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:41.781942 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:49:41.782697 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:41.782710 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.784101 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.784106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:41.785022 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:41.785035 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.785540 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.785545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:41.785690 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:41.786328 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:49:41.786348 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:49:41.788011 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:49:41.790079 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:41.790082 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:41.790084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:41.790348 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:41.790686 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:41.790688 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:41.790690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:41.790778 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:41.791233 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:49:41.791235 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:49:41.791336 (+ 101us) client_negotiation.cc:769] Sending connection context
0810 01:49:41.791422 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 01:49:41.791554 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260810 01:49:41.791846 30728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.780982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43718 (local address 127.9.77.189:40369)
0810 01:49:41.781170 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:41.781176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:41.781266 (+ 90us) server_negotiation.cc:407] Connection header received
0810 01:49:41.781474 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:41.781476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:41.781530 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:41.781609 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:41.782848 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.783956 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.785152 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.785416 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.785502 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:41.788229 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:41.788235 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:41.788240 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:41.788280 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:41.789964 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:41.790479 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:41.790485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:41.790488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:41.790591 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:41.790869 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:41.790875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:41.790878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:41.791138 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:41.791245 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:41.791512 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 01:49:41.791663 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 01:49:41.792843 30600 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } has no permanent_uuid. Determining permanent_uuid...
I20260810 01:49:41.793686 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.793425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:39015 (local address 127.0.0.1:36210)
0810 01:49:41.793561 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.793622 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:39015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 01:49:41.794409 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.794195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:39015 (local address 127.0.0.1:36220)
0810 01:49:41.794298 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.794346 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:39015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 01:49:41.794565 30600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.188:39015: Network error: Client connection negotiation failed: client connection to 127.9.77.188:39015: connect: Connection refused (error 111)
I20260810 01:49:41.820932 30718 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } attempt: 2
I20260810 01:49:41.822099 30724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.821736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:39015 (local address 127.0.0.1:36236)
0810 01:49:41.821924 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.821998 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:39015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 01:49:41.822912 30724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.822701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:39015 (local address 127.0.0.1:36248)
0810 01:49:41.822805 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.822851 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:39015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 01:49:41.823072 30718 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.188:39015: Network error: Client connection negotiation failed: client connection to 127.9.77.188:39015: connect: Connection refused (error 111)
W20260810 01:49:41.847317 30732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:49:41.847719 30732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:49:41.847766 30732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:49:41.852370 30732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:49:41.852444 30732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:49:41.852464 30732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:49:41.852483 30732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:49:41.852499 30732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:49:41.853727 30600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } attempt: 1
I20260810 01:49:41.854775 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.854408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:39015 (local address 127.0.0.1:36256)
0810 01:49:41.854585 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.854671 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:39015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 01:49:41.855650 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.855383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:39015 (local address 127.0.0.1:36260)
0810 01:49:41.855518 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.855576 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:39015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 01:49:41.855805 30600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.188:39015: Network error: Client connection negotiation failed: client connection to 127.9.77.188:39015: connect: Connection refused (error 111)
I20260810 01:49:41.858664 30732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:37017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:45681,127.9.77.189:40369,127.9.77.188:39015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.188.keytab
--principal=kudu/127.9.77.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.188:39015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.77.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:49:41.860296 30732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:49:41.861722 30732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:49:41.872076 30739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:41.872049 30744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:41.872291 30732 server_base.cc:1034] running on GCE node
W20260810 01:49:41.872004 30742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:41.873253 30732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:49:41.874687 30732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:49:41.875890 30732 hybrid_clock.cc:648] HybridClock initialized: now 1786326581875869 us; error 37 us; skew 500 ppm
Aug 10 01:49:41 dist-test-slave-kvfs krb5kdc[30471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326581, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:49:41.879387 30732 init.cc:377] Logged in from keytab as kudu/127.9.77.188@KRBTEST.COM (short username kudu)
I20260810 01:49:41.880637 30732 webserver.cc:460] Webserver started at http://127.9.77.188:43695/ using document root <none> and password file <none>
I20260810 01:49:41.881538 30732 fs_manager.cc:359] Metadata directory not provided
I20260810 01:49:41.881655 30732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:49:41.881989 30732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:49:41.885391 30732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-2/data/instance:
uuid: "489061a0a9044f58a4f20c5bcd72252d"
format_stamp: "Formatted at 2026-08-10 01:49:41 on dist-test-slave-kvfs"
server_key: "6d99b05d2b4bd22b702c0c9b5a475480"
server_key_iv: "542295db92001e549d91842b24f180c3"
server_key_version: "encryptionkey@0"
I20260810 01:49:41.886089 30732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-2/wal/instance:
uuid: "489061a0a9044f58a4f20c5bcd72252d"
format_stamp: "Formatted at 2026-08-10 01:49:41 on dist-test-slave-kvfs"
server_key: "6d99b05d2b4bd22b702c0c9b5a475480"
server_key_iv: "542295db92001e549d91842b24f180c3"
server_key_version: "encryptionkey@0"
I20260810 01:49:41.890865 30732 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 01:49:41.894024 30753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:41.895500 30732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 01:49:41.895664 30732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-2/wal
uuid: "489061a0a9044f58a4f20c5bcd72252d"
format_stamp: "Formatted at 2026-08-10 01:49:41 on dist-test-slave-kvfs"
server_key: "6d99b05d2b4bd22b702c0c9b5a475480"
server_key_iv: "542295db92001e549d91842b24f180c3"
server_key_version: "encryptionkey@0"
I20260810 01:49:41.895804 30732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:49:41.914270 30718 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } attempt: 3
I20260810 01:49:41.915380 30724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.915013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:39015 (local address 127.0.0.1:36262)
0810 01:49:41.915199 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.915271 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:39015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 01:49:41.916270 30724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.915966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:39015 (local address 127.0.0.1:36268)
0810 01:49:41.916148 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.916206 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:39015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260810 01:49:41.916399 30718 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.188:39015: Network error: Client connection negotiation failed: client connection to 127.9.77.188:39015: connect: Connection refused (error 111)
I20260810 01:49:41.917068 30732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:49:41.920900 30732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:49:41.921145 30732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:49:41.922971 30600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } attempt: 2
I20260810 01:49:41.923861 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.923542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:39015 (local address 127.0.0.1:36282)
0810 01:49:41.923696 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.923769 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:39015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 01:49:41.924628 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.924393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:39015 (local address 127.0.0.1:36298)
0810 01:49:41.924511 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.924565 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:39015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 01:49:41.924764 30600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.188:39015: Network error: Client connection negotiation failed: client connection to 127.9.77.188:39015: connect: Connection refused (error 111)
I20260810 01:49:41.930353 30732 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.188:39015
I20260810 01:49:41.930388 30832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.188:39015 every 8 connection(s)
I20260810 01:49:41.931767 30732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-2/data/info.pb
I20260810 01:49:41.933300 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 30732
I20260810 01:49:41.933496 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786326392877349-9526-0/minicluster-data/master-2/wal/instance
I20260810 01:49:41.933847 9526 external_mini_cluster.cc:1428] Setting key 47b39a770161f8015a0626b1706d7eaa
I20260810 01:49:41.936105 30833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:49:41.941949 30833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45681 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 01:49:41 dist-test-slave-kvfs krb5kdc[30471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.188@KRBTEST.COM
Aug 10 01:49:41 dist-test-slave-kvfs krb5kdc[30471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326581, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.188@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:49:41.952524 30837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.935653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36310 (local address 127.9.77.188:39015)
0810 01:49:41.936225 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:41.936249 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:41.936301 (+ 52us) server_negotiation.cc:407] Connection header received
0810 01:49:41.937307 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:41.937337 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:41.937850 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:41.938290 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:41.939147 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.940818 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.941774 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.942557 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.942727 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:41.946296 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:41.946305 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:41.946319 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:41.946362 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:41.948762 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:41.949326 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:41.949333 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:41.949340 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:41.949503 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:41.949905 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:41.949919 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:41.949923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:41.950561 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:41.950808 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:41.951089 (+ 281us) server_negotiation.cc:299] Negotiation successful
0810 01:49:41.951425 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":166,"threads_started":1}
I20260810 01:49:41.955750 30606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.944680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50314 (local address 127.9.77.190:45681)
0810 01:49:41.944875 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:41.944881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:41.945313 (+ 432us) server_negotiation.cc:407] Connection header received
0810 01:49:41.945501 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:41.945504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:41.945588 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:41.945687 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:41.946730 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.947399 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.948863 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.949117 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.949206 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:41.952084 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:41.952088 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:41.952090 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:41.952116 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:41.953781 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:41.954268 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:41.954271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:41.954273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:41.954372 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:41.954648 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:41.954654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:41.954657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:41.954954 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:41.955036 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:41.955448 (+ 412us) server_negotiation.cc:299] Negotiation successful
0810 01:49:41.955593 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 01:49:41.955790 30838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.944480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45681 (local address 127.0.0.1:50314)
0810 01:49:41.945099 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.945173 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:41.945380 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:41.945775 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:41.945780 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:41.946314 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:49:41.946546 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:41.946559 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.947617 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.947622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:41.948701 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:41.948714 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.949240 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.949247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:41.949375 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:41.950137 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:49:41.950155 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:49:41.951891 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:49:41.953905 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:41.953912 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:41.953917 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:41.954164 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:41.954474 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:41.954476 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:41.954477 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:41.954557 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:41.955041 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:49:41.955047 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:49:41.955177 (+ 130us) client_negotiation.cc:769] Sending connection context
0810 01:49:41.955352 (+ 175us) client_negotiation.cc:240] Negotiation successful
0810 01:49:41.955562 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":229,"threads_started":1}
I20260810 01:49:41.956934 30833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40369 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 01:49:41 dist-test-slave-kvfs krb5kdc[30471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
Aug 10 01:49:41 dist-test-slave-kvfs krb5kdc[30471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326581, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.188@KRBTEST.COM for kudu/127.9.77.189@KRBTEST.COM
I20260810 01:49:41.966260 30839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.955533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50318 (local address 127.9.77.190:45681)
0810 01:49:41.955877 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:41.955894 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:41.956460 (+ 566us) server_negotiation.cc:407] Connection header received
0810 01:49:41.956620 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:41.956623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:41.956679 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:41.956777 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:41.957591 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.958236 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.959153 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.959461 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.959543 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:41.962470 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:41.962473 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:41.962475 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:41.962521 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:41.964265 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:41.964761 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:41.964764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:41.964766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:41.964864 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:41.965168 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:41.965171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:41.965172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:41.965431 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:41.965668 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:41.965850 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 01:49:41.965992 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":114,"threads_started":1}
I20260810 01:49:41.966485 30838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.957550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:40369 (local address 127.0.0.1:43730)
0810 01:49:41.957650 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.957665 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:41.957737 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:41.958126 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:41.958130 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:41.958314 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:49:41.958432 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:41.958448 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.959212 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.959215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:41.959924 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:41.959935 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.960465 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.960468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:41.960581 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:41.961046 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:49:41.961058 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:49:41.962735 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:49:41.964761 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:41.964764 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:41.964766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:41.965038 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:41.965341 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:41.965344 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:41.965346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:41.965436 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:41.965887 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:49:41.965891 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:49:41.966002 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 01:49:41.966147 (+ 145us) client_negotiation.cc:240] Negotiation successful
0810 01:49:41.966288 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 01:49:41.966493 30728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.957720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43730 (local address 127.9.77.189:40369)
0810 01:49:41.957899 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:41.957903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:41.957916 (+ 13us) server_negotiation.cc:407] Connection header received
0810 01:49:41.957970 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:41.957972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:41.958017 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:41.958114 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:41.958571 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.959068 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.960092 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.960336 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.960459 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:41.962868 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:41.962874 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:41.962878 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:41.962933 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:41.964654 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:41.965138 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:41.965145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:41.965148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:41.965243 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:41.965543 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:41.965546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:41.965548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:41.965788 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:41.965879 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:41.966247 (+ 368us) server_negotiation.cc:299] Negotiation successful
0810 01:49:41.966356 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 01:49:41.967474 30833 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 01:49:41 dist-test-slave-kvfs krb5kdc[30471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326581, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.188@KRBTEST.COM for kudu/127.9.77.188@KRBTEST.COM
Aug 10 01:49:41 dist-test-slave-kvfs krb5kdc[30471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.189@KRBTEST.COM
I20260810 01:49:41.977483 30838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.968108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:39015 (local address 127.0.0.1:36318)
0810 01:49:41.968226 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:41.968240 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:41.968316 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:41.968640 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:41.968642 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:41.968773 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:49:41.968845 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:41.968850 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.970239 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.970241 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:41.970892 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:41.970901 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.971524 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.971530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:41.971635 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:41.972104 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:49:41.972117 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:49:41.973459 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:49:41.975450 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:41.975454 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:41.975456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:41.975723 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:41.976159 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:41.976163 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:41.976167 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:41.976276 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:41.976900 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:49:41.976903 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:49:41.977011 (+ 108us) client_negotiation.cc:769] Sending connection context
0810 01:49:41.977127 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 01:49:41.977266 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 01:49:41.977592 30837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.968287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36318 (local address 127.9.77.188:39015)
0810 01:49:41.968412 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:41.968415 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:41.968425 (+ 10us) server_negotiation.cc:407] Connection header received
0810 01:49:41.968475 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:41.968478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:41.968522 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:41.968609 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:41.968964 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.970125 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.971041 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.971407 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.971491 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:41.973594 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:41.973597 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:41.973599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:41.973620 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:41.975316 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:41.975857 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:41.975860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:41.975862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:41.975960 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:41.976434 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:41.976437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:41.976439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:41.976773 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:41.976889 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:41.977244 (+ 355us) server_negotiation.cc:299] Negotiation successful
0810 01:49:41.977408 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 01:49:41.978749 30728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.968097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43736 (local address 127.9.77.189:40369)
0810 01:49:41.968281 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:41.968286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:41.968302 (+ 16us) server_negotiation.cc:407] Connection header received
0810 01:49:41.968337 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:41.968343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:41.968395 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:41.968479 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:41.969205 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.969707 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.970606 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.970944 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.971044 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:41.974294 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:41.974298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:41.974301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:41.974327 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:41.976302 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:41.976906 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:41.976910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:41.976912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:41.977012 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:41.977412 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:41.977415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:41.977417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:41.978057 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:41.978362 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:41.978484 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 01:49:41.978597 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260810 01:49:41.982168 30833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 489061a0a9044f58a4f20c5bcd72252d: Bootstrap starting.
I20260810 01:49:41.982821 9526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 01:49:41.985939 30833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 489061a0a9044f58a4f20c5bcd72252d: Neither blocks nor log segments found. Creating new log.
I20260810 01:49:41.987084 30833 log.cc:824] T 00000000000000000000000000000000 P 489061a0a9044f58a4f20c5bcd72252d: Log is configured to *not* fsync() on all Append() calls
I20260810 01:49:41.990010 30833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 489061a0a9044f58a4f20c5bcd72252d: No bootstrap required, opened a new log
I20260810 01:49:41.995790 30833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 489061a0a9044f58a4f20c5bcd72252d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf9a150e9ef4db5ab28be9ede40fe90" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45681 } } peers { permanent_uuid: "6b992eba730945beb0fb5ee818bf73d6" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40369 } } peers { permanent_uuid: "489061a0a9044f58a4f20c5bcd72252d" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } }
I20260810 01:49:41.996184 30833 raft_consensus.cc:374] T 00000000000000000000000000000000 P 489061a0a9044f58a4f20c5bcd72252d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:49:41.996255 30833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 489061a0a9044f58a4f20c5bcd72252d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 489061a0a9044f58a4f20c5bcd72252d, State: Initialized, Role: FOLLOWER
I20260810 01:49:41.996901 30833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 489061a0a9044f58a4f20c5bcd72252d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf9a150e9ef4db5ab28be9ede40fe90" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45681 } } peers { permanent_uuid: "6b992eba730945beb0fb5ee818bf73d6" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40369 } } peers { permanent_uuid: "489061a0a9044f58a4f20c5bcd72252d" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } }
I20260810 01:49:41.997393 30728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.985677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43752 (local address 127.9.77.189:40369)
0810 01:49:41.986014 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:41.986019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:41.986579 (+ 560us) server_negotiation.cc:407] Connection header received
0810 01:49:41.986665 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:41.986668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:41.986714 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:41.987660 (+ 946us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:41.987774 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.988605 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.990116 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.990456 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.990842 (+ 386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:41.992896 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:41.992908 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:41.992911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:41.992935 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:41.994834 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:41.995533 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:41.995540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:41.995545 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:41.995644 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:41.996009 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:41.996012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:41.996014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:41.996262 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:41.996986 (+ 724us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:41.997072 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 01:49:41.997210 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253}
I20260810 01:49:41.997637 30839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.985730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50332 (local address 127.9.77.190:45681)
0810 01:49:41.985894 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:41.985910 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:41.986121 (+ 211us) server_negotiation.cc:407] Connection header received
0810 01:49:41.986426 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:41.986429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:41.986504 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:41.986572 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:41.987705 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.988634 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.990117 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.990496 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.991146 (+ 650us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:41.992896 (+ 1750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:41.992899 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:41.992902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:41.992935 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:41.994981 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:41.995593 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:41.995599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:41.995602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:41.995687 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:41.995987 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:41.995992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:41.995994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:41.996262 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:41.997235 (+ 973us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:41.997313 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 01:49:41.997458 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 01:49:41.998298 30837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:41.986190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36332 (local address 127.9.77.188:39015)
0810 01:49:41.986319 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:41.986323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:41.986870 (+ 547us) server_negotiation.cc:407] Connection header received
0810 01:49:41.987061 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:41.987068 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:41.987125 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:41.987185 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:41.988263 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.988961 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.990195 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:41.990519 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:41.990594 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:41.992967 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:41.992971 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:41.992974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:41.992999 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:41.995482 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:41.995994 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:41.995998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:41.996000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:41.996111 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:41.996653 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:41.996656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:41.996658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:41.996919 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:41.997023 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:41.998031 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0810 01:49:41.998156 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 01:49:41.998795 30833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 489061a0a9044f58a4f20c5bcd72252d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:49:41.999217 30848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 489061a0a9044f58a4f20c5bcd72252d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf9a150e9ef4db5ab28be9ede40fe90" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45681 } } peers { permanent_uuid: "6b992eba730945beb0fb5ee818bf73d6" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40369 } } peers { permanent_uuid: "489061a0a9044f58a4f20c5bcd72252d" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } } }
I20260810 01:49:41.999363 30848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 489061a0a9044f58a4f20c5bcd72252d [sys.catalog]: This master's current role is: FOLLOWER
W20260810 01:49:42.004916 30865 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 489061a0a9044f58a4f20c5bcd72252d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 01:49:42.006438 30865 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 01:49:42.002758 30732 master_runner.cc:186] Error getting master registration for 127.9.77.190:45681: OK, instance_id { permanent_uuid: "4bf9a150e9ef4db5ab28be9ede40fe90" instance_seqno: 1786326581382335 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 01:49:42.006620 30732 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 01:49:42.024955 30600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } attempt: 3
Aug 10 01:49:42 dist-test-slave-kvfs krb5kdc[30471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326581, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for kudu/127.9.77.188@KRBTEST.COM
I20260810 01:49:42.035530 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:42.025518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:39015 (local address 127.0.0.1:36346)
0810 01:49:42.025725 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:42.025762 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:42.025849 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:42.026132 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:42.026134 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:42.026369 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:49:42.026487 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:42.026493 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:42.027594 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:42.027597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:42.028285 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:42.028295 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:42.028878 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:42.028881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:42.028953 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:42.029657 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:49:42.029671 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:49:42.031590 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:49:42.033836 (+ 2246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:42.033839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:42.033841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:42.034067 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:42.034377 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:42.034381 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:42.034382 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:42.034480 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:42.034916 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:49:42.034919 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:49:42.035038 (+ 119us) client_negotiation.cc:769] Sending connection context
0810 01:49:42.035151 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 01:49:42.035335 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260810 01:49:42.035530 30837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:42.025633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36346 (local address 127.9.77.188:39015)
0810 01:49:42.025791 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:42.025796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:42.025813 (+ 17us) server_negotiation.cc:407] Connection header received
0810 01:49:42.025954 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:42.025958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:42.026009 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:42.026093 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:42.026606 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:42.027476 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:42.028398 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:42.028752 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:42.028849 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:42.031776 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:42.031780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:42.031783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:42.031811 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:42.033659 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:42.034173 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:42.034177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:42.034179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:42.034273 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:42.034583 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:42.034586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:42.034588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:42.034813 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:42.034929 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:42.035215 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 01:49:42.035349 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 01:49:42.040007 30600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bf9a150e9ef4db5ab28be9ede40fe90: Bootstrap starting.
I20260810 01:49:42.043216 30600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bf9a150e9ef4db5ab28be9ede40fe90: Neither blocks nor log segments found. Creating new log.
I20260810 01:49:42.044212 30600 log.cc:824] T 00000000000000000000000000000000 P 4bf9a150e9ef4db5ab28be9ede40fe90: Log is configured to *not* fsync() on all Append() calls
I20260810 01:49:42.046504 30600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bf9a150e9ef4db5ab28be9ede40fe90: No bootstrap required, opened a new log
I20260810 01:49:42.051156 30600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bf9a150e9ef4db5ab28be9ede40fe90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf9a150e9ef4db5ab28be9ede40fe90" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45681 } } peers { permanent_uuid: "6b992eba730945beb0fb5ee818bf73d6" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40369 } } peers { permanent_uuid: "489061a0a9044f58a4f20c5bcd72252d" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } }
I20260810 01:49:42.051378 30600 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bf9a150e9ef4db5ab28be9ede40fe90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:49:42.051452 30600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bf9a150e9ef4db5ab28be9ede40fe90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bf9a150e9ef4db5ab28be9ede40fe90, State: Initialized, Role: FOLLOWER
I20260810 01:49:42.051960 30600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bf9a150e9ef4db5ab28be9ede40fe90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf9a150e9ef4db5ab28be9ede40fe90" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45681 } } peers { permanent_uuid: "6b992eba730945beb0fb5ee818bf73d6" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40369 } } peers { permanent_uuid: "489061a0a9044f58a4f20c5bcd72252d" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } }
I20260810 01:49:42.052598 30869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bf9a150e9ef4db5ab28be9ede40fe90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf9a150e9ef4db5ab28be9ede40fe90" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45681 } } peers { permanent_uuid: "6b992eba730945beb0fb5ee818bf73d6" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40369 } } peers { permanent_uuid: "489061a0a9044f58a4f20c5bcd72252d" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } } }
I20260810 01:49:42.052764 30869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bf9a150e9ef4db5ab28be9ede40fe90 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 01:49:42.053611 30600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bf9a150e9ef4db5ab28be9ede40fe90 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:49:42.059551 30718 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } attempt: 4
W20260810 01:49:42.061139 30888 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4bf9a150e9ef4db5ab28be9ede40fe90: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 01:49:42.061301 30888 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 01:49:42.059078 30491 master_runner.cc:186] Error getting master registration for 127.9.77.189:40369: OK, instance_id { permanent_uuid: "6b992eba730945beb0fb5ee818bf73d6" instance_seqno: 1786326581653101 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 01:49:42.062781 30491 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 10 01:49:42 dist-test-slave-kvfs krb5kdc[30471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326581, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.189@KRBTEST.COM for kudu/127.9.77.188@KRBTEST.COM
I20260810 01:49:42.071310 30724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:42.060132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:39015 (local address 127.0.0.1:36354)
0810 01:49:42.060352 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:42.060368 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:42.060456 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:42.062825 (+ 2369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:42.062830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:42.063073 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:49:42.063185 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:42.063191 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:42.064258 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:42.064261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:42.064919 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:42.064930 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:42.065415 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:42.065418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:42.065501 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:42.066117 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:49:42.066130 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:49:42.067673 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:49:42.069628 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:42.069633 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:42.069636 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:42.069868 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:42.070167 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:42.070171 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:42.070173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:42.070271 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:42.070692 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:49:42.070695 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:49:42.070822 (+ 127us) client_negotiation.cc:769] Sending connection context
0810 01:49:42.070936 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 01:49:42.071107 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260810 01:49:42.071310 30837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:42.061653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36354 (local address 127.9.77.188:39015)
0810 01:49:42.062459 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:42.062464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:42.062478 (+ 14us) server_negotiation.cc:407] Connection header received
0810 01:49:42.062519 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:42.062522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:42.062574 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:42.062660 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:42.063311 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:42.064124 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:42.065052 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:42.065297 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:42.065415 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:42.067825 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:42.067829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:42.067832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:42.067859 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:42.069507 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:42.069973 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:42.069978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:42.069980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:42.070071 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:42.070374 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:42.070377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:42.070379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:42.070593 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:42.070705 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:42.071004 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 01:49:42.071130 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709}
I20260810 01:49:42.075055 30718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6: Bootstrap starting.
I20260810 01:49:42.078572 30718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6: Neither blocks nor log segments found. Creating new log.
I20260810 01:49:42.079586 30718 log.cc:824] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6: Log is configured to *not* fsync() on all Append() calls
I20260810 01:49:42.082046 30718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6: No bootstrap required, opened a new log
I20260810 01:49:42.085922 30718 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf9a150e9ef4db5ab28be9ede40fe90" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45681 } } peers { permanent_uuid: "6b992eba730945beb0fb5ee818bf73d6" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40369 } } peers { permanent_uuid: "489061a0a9044f58a4f20c5bcd72252d" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } }
I20260810 01:49:42.086174 30718 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:49:42.086269 30718 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b992eba730945beb0fb5ee818bf73d6, State: Initialized, Role: FOLLOWER
I20260810 01:49:42.086863 30718 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf9a150e9ef4db5ab28be9ede40fe90" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45681 } } peers { permanent_uuid: "6b992eba730945beb0fb5ee818bf73d6" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40369 } } peers { permanent_uuid: "489061a0a9044f58a4f20c5bcd72252d" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } }
I20260810 01:49:42.087513 30891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf9a150e9ef4db5ab28be9ede40fe90" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45681 } } peers { permanent_uuid: "6b992eba730945beb0fb5ee818bf73d6" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40369 } } peers { permanent_uuid: "489061a0a9044f58a4f20c5bcd72252d" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } } }
I20260810 01:49:42.087719 30891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 01:49:42.088635 30718 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 01:49:42.097419 30908 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 01:49:42.097625 30908 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 01:49:42.098491 30610 master_runner.cc:263] No leader master found from master 6b992eba730945beb0fb5ee818bf73d6
I20260810 01:49:42.098562 30610 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 01:49:42.127836 30522 tablet.cc:2369] T 00000000000000000000000000000000 P 4bf9a150e9ef4db5ab28be9ede40fe90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 01:49:42.128786 30891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:49:42.129005 30891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf9a150e9ef4db5ab28be9ede40fe90" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45681 } } peers { permanent_uuid: "6b992eba730945beb0fb5ee818bf73d6" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40369 } } peers { permanent_uuid: "489061a0a9044f58a4f20c5bcd72252d" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } }
I20260810 01:49:42.130245 30891 leader_election.cc:288] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bf9a150e9ef4db5ab28be9ede40fe90 (127.9.77.190:45681), 489061a0a9044f58a4f20c5bcd72252d (127.9.77.188:39015)
I20260810 01:49:42.131150 30557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b992eba730945beb0fb5ee818bf73d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bf9a150e9ef4db5ab28be9ede40fe90" is_pre_election: true
I20260810 01:49:42.131287 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b992eba730945beb0fb5ee818bf73d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "489061a0a9044f58a4f20c5bcd72252d" is_pre_election: true
I20260810 01:49:42.131533 30557 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4bf9a150e9ef4db5ab28be9ede40fe90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b992eba730945beb0fb5ee818bf73d6 in term 0.
I20260810 01:49:42.131662 30788 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 489061a0a9044f58a4f20c5bcd72252d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b992eba730945beb0fb5ee818bf73d6 in term 0.
I20260810 01:49:42.131979 30633 leader_election.cc:302] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bf9a150e9ef4db5ab28be9ede40fe90, 6b992eba730945beb0fb5ee818bf73d6; no voters:
I20260810 01:49:42.132624 30891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:49:42.132720 30891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:49:42.132764 30891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:49:42.134452 30891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf9a150e9ef4db5ab28be9ede40fe90" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45681 } } peers { permanent_uuid: "6b992eba730945beb0fb5ee818bf73d6" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40369 } } peers { permanent_uuid: "489061a0a9044f58a4f20c5bcd72252d" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } }
I20260810 01:49:42.134857 30891 leader_election.cc:288] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6 [CANDIDATE]: Term 1 election: Requested vote from peers 4bf9a150e9ef4db5ab28be9ede40fe90 (127.9.77.190:45681), 489061a0a9044f58a4f20c5bcd72252d (127.9.77.188:39015)
I20260810 01:49:42.135315 30557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b992eba730945beb0fb5ee818bf73d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bf9a150e9ef4db5ab28be9ede40fe90"
I20260810 01:49:42.135361 30788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b992eba730945beb0fb5ee818bf73d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "489061a0a9044f58a4f20c5bcd72252d"
I20260810 01:49:42.135461 30557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bf9a150e9ef4db5ab28be9ede40fe90 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:49:42.135504 30788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 489061a0a9044f58a4f20c5bcd72252d [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:49:42.137063 30557 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4bf9a150e9ef4db5ab28be9ede40fe90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b992eba730945beb0fb5ee818bf73d6 in term 1.
I20260810 01:49:42.137097 30788 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 489061a0a9044f58a4f20c5bcd72252d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b992eba730945beb0fb5ee818bf73d6 in term 1.
I20260810 01:49:42.137450 30633 leader_election.cc:302] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bf9a150e9ef4db5ab28be9ede40fe90, 6b992eba730945beb0fb5ee818bf73d6; no voters:
I20260810 01:49:42.137696 30891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:49:42.137938 30891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6 [term 1 LEADER]: Becoming Leader. State: Replica: 6b992eba730945beb0fb5ee818bf73d6, State: Running, Role: LEADER
I20260810 01:49:42.138401 30891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf9a150e9ef4db5ab28be9ede40fe90" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45681 } } peers { permanent_uuid: "6b992eba730945beb0fb5ee818bf73d6" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40369 } } peers { permanent_uuid: "489061a0a9044f58a4f20c5bcd72252d" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } }
I20260810 01:49:42.140690 30891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b992eba730945beb0fb5ee818bf73d6. Latest consensus state: current_term: 1 leader_uuid: "6b992eba730945beb0fb5ee818bf73d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf9a150e9ef4db5ab28be9ede40fe90" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45681 } } peers { permanent_uuid: "6b992eba730945beb0fb5ee818bf73d6" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40369 } } peers { permanent_uuid: "489061a0a9044f58a4f20c5bcd72252d" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } } }
I20260810 01:49:42.140885 30891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6 [sys.catalog]: This master's current role is: LEADER
I20260810 01:49:42.143477 30912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:49:42.147879 30912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260810 01:49:42.148964 30638 tablet.cc:2369] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 01:49:42.155191 30788 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 489061a0a9044f58a4f20c5bcd72252d [term 1 FOLLOWER]: Refusing update from remote peer 6b992eba730945beb0fb5ee818bf73d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:49:42.155689 30557 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4bf9a150e9ef4db5ab28be9ede40fe90 [term 1 FOLLOWER]: Refusing update from remote peer 6b992eba730945beb0fb5ee818bf73d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:49:42.156329 30910 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "489061a0a9044f58a4f20c5bcd72252d" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:49:42.156570 30891 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bf9a150e9ef4db5ab28be9ede40fe90" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:49:42.162011 30848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 489061a0a9044f58a4f20c5bcd72252d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b992eba730945beb0fb5ee818bf73d6. Latest consensus state: current_term: 1 leader_uuid: "6b992eba730945beb0fb5ee818bf73d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf9a150e9ef4db5ab28be9ede40fe90" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45681 } } peers { permanent_uuid: "6b992eba730945beb0fb5ee818bf73d6" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40369 } } peers { permanent_uuid: "489061a0a9044f58a4f20c5bcd72252d" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } } }
I20260810 01:49:42.162014 30869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bf9a150e9ef4db5ab28be9ede40fe90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b992eba730945beb0fb5ee818bf73d6. Latest consensus state: current_term: 1 leader_uuid: "6b992eba730945beb0fb5ee818bf73d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf9a150e9ef4db5ab28be9ede40fe90" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45681 } } peers { permanent_uuid: "6b992eba730945beb0fb5ee818bf73d6" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40369 } } peers { permanent_uuid: "489061a0a9044f58a4f20c5bcd72252d" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } } }
I20260810 01:49:42.162179 30869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bf9a150e9ef4db5ab28be9ede40fe90 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 01:49:42.162181 30848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 489061a0a9044f58a4f20c5bcd72252d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 01:49:42.166072 30910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b992eba730945beb0fb5ee818bf73d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf9a150e9ef4db5ab28be9ede40fe90" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45681 } } peers { permanent_uuid: "6b992eba730945beb0fb5ee818bf73d6" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40369 } } peers { permanent_uuid: "489061a0a9044f58a4f20c5bcd72252d" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } } }
I20260810 01:49:42.166229 30910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6 [sys.catalog]: This master's current role is: LEADER
I20260810 01:49:42.166982 30914 mvcc.cc:205] Tried to move back new op lower bound from 7316793680497303552 to 7316793680443113472. Current Snapshot: MvccSnapshot[applied={T|T < 7316793680497303552}]
I20260810 01:49:42.167819 30848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 489061a0a9044f58a4f20c5bcd72252d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b992eba730945beb0fb5ee818bf73d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf9a150e9ef4db5ab28be9ede40fe90" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45681 } } peers { permanent_uuid: "6b992eba730945beb0fb5ee818bf73d6" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40369 } } peers { permanent_uuid: "489061a0a9044f58a4f20c5bcd72252d" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } } }
I20260810 01:49:42.167944 30848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 489061a0a9044f58a4f20c5bcd72252d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 01:49:42.171545 30916 mvcc.cc:205] Tried to move back new op lower bound from 7316793680497303552 to 7316793680443113472. Current Snapshot: MvccSnapshot[applied={T|T < 7316793680497303552}]
I20260810 01:49:42.177798 30891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b992eba730945beb0fb5ee818bf73d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf9a150e9ef4db5ab28be9ede40fe90" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45681 } } peers { permanent_uuid: "6b992eba730945beb0fb5ee818bf73d6" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40369 } } peers { permanent_uuid: "489061a0a9044f58a4f20c5bcd72252d" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } } }
I20260810 01:49:42.177991 30891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6 [sys.catalog]: This master's current role is: LEADER
I20260810 01:49:42.178278 30912 catalog_manager.cc:1347] Generated new cluster ID: 0fadaceac6f84f7aa6a03326fdda307f
I20260810 01:49:42.178347 30912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:49:42.179512 30869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bf9a150e9ef4db5ab28be9ede40fe90 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b992eba730945beb0fb5ee818bf73d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf9a150e9ef4db5ab28be9ede40fe90" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45681 } } peers { permanent_uuid: "6b992eba730945beb0fb5ee818bf73d6" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40369 } } peers { permanent_uuid: "489061a0a9044f58a4f20c5bcd72252d" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 39015 } } }
I20260810 01:49:42.179658 30869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bf9a150e9ef4db5ab28be9ede40fe90 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 01:49:42.180578 30917 mvcc.cc:205] Tried to move back new op lower bound from 7316793680497303552 to 7316793680443113472. Current Snapshot: MvccSnapshot[applied={T|T < 7316793680497303552}]
I20260810 01:49:42.204371 30912 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:49:42.206094 30912 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:49:42.227613 30912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b992eba730945beb0fb5ee818bf73d6: Generated new TSK 0
I20260810 01:49:42.229091 30912 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:49:43.008442 30865 catalog_manager.cc:1259] Loaded cluster ID: 0fadaceac6f84f7aa6a03326fdda307f
I20260810 01:49:43.008561 30865 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 489061a0a9044f58a4f20c5bcd72252d: loading cluster ID for follower catalog manager: success
I20260810 01:49:43.010617 30865 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 489061a0a9044f58a4f20c5bcd72252d: acquiring CA information for follower catalog manager: success
I20260810 01:49:43.012193 30865 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 489061a0a9044f58a4f20c5bcd72252d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 01:49:43.063294 30888 catalog_manager.cc:1259] Loaded cluster ID: 0fadaceac6f84f7aa6a03326fdda307f
I20260810 01:49:43.063474 30888 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4bf9a150e9ef4db5ab28be9ede40fe90: loading cluster ID for follower catalog manager: success
I20260810 01:49:43.065562 30888 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4bf9a150e9ef4db5ab28be9ede40fe90: acquiring CA information for follower catalog manager: success
I20260810 01:49:43.067102 30888 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4bf9a150e9ef4db5ab28be9ede40fe90: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 01:49:43.299957 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 30491
I20260810 01:49:43.317104 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 30610
I20260810 01:49:43.336215 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 30732
2026-08-10T01:49:43Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3627 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:49:43 dist-test-slave-kvfs krb5kdc[30938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:49:43 dist-test-slave-kvfs krb5kdc[30938](info): set up 2 sockets
Aug 10 01:49:43 dist-test-slave-kvfs krb5kdc[30938](info): commencing operation
krb5kdc: starting...
W20260810 01:49:44.641328 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.272s user 0.004s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:49:44 dist-test-slave-kvfs krb5kdc[30938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:49:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:49:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:49:44.795948 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:33749
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:45631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:33749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:49:44.940932 30959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:49:44.941340 30959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:49:44.941392 30959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:49:44.945981 30959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:49:44.946069 30959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:49:44.946090 30959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:49:44.946110 30959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:49:44.946125 30959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:49:44.952291 30959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:45631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:33749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:33749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:49:44.953890 30959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:49:44.955376 30959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:49:44.966511 30966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:44.966511 30967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:44.966511 30969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:44.966631 30959 server_base.cc:1034] running on GCE node
I20260810 01:49:44.967772 30959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:49:44.969426 30959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:49:44.970662 30959 hybrid_clock.cc:648] HybridClock initialized: now 1786326584970644 us; error 42 us; skew 500 ppm
Aug 10 01:49:44 dist-test-slave-kvfs krb5kdc[30938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326584, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:49:44.979035 30959 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:49:44.980400 30959 webserver.cc:460] Webserver started at http://127.9.77.190:35939/ using document root <none> and password file <none>
I20260810 01:49:44.981159 30959 fs_manager.cc:359] Metadata directory not provided
I20260810 01:49:44.981232 30959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:49:44.981534 30959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:49:44.984356 30959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "f9bc943de35749e1ac40b04b34a44f6c"
format_stamp: "Formatted at 2026-08-10 01:49:44 on dist-test-slave-kvfs"
server_key: "3c708f90b56e17c4b532e3ad4e819809"
server_key_iv: "3613c09cbfbdcb917dac13e5810fda54"
server_key_version: "encryptionkey@0"
I20260810 01:49:44.985081 30959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "f9bc943de35749e1ac40b04b34a44f6c"
format_stamp: "Formatted at 2026-08-10 01:49:44 on dist-test-slave-kvfs"
server_key: "3c708f90b56e17c4b532e3ad4e819809"
server_key_iv: "3613c09cbfbdcb917dac13e5810fda54"
server_key_version: "encryptionkey@0"
I20260810 01:49:44.989889 30959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 01:49:44.993010 30979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:44.994514 30959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:49:44.994679 30959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "f9bc943de35749e1ac40b04b34a44f6c"
format_stamp: "Formatted at 2026-08-10 01:49:44 on dist-test-slave-kvfs"
server_key: "3c708f90b56e17c4b532e3ad4e819809"
server_key_iv: "3613c09cbfbdcb917dac13e5810fda54"
server_key_version: "encryptionkey@0"
I20260810 01:49:44.994828 30959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:49:45.011021 30959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:49:45.015012 30959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:49:45.015268 30959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:49:45.024061 30959 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:33749
I20260810 01:49:45.024080 31064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:33749 every 8 connection(s)
I20260810 01:49:45.025280 30959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:49:45.026335 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 30959
I20260810 01:49:45.026453 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:49:45.026820 9526 external_mini_cluster.cc:1428] Setting key 165aa5ba9f443dee9f18c98764abb223
I20260810 01:49:45.029367 31066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:49:45.037796 31066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9bc943de35749e1ac40b04b34a44f6c: Bootstrap starting.
Aug 10 01:49:45 dist-test-slave-kvfs krb5kdc[30938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:49:45.041400 31066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9bc943de35749e1ac40b04b34a44f6c: Neither blocks nor log segments found. Creating new log.
I20260810 01:49:45.042390 31066 log.cc:824] T 00000000000000000000000000000000 P f9bc943de35749e1ac40b04b34a44f6c: Log is configured to *not* fsync() on all Append() calls
I20260810 01:49:45.044584 31070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:45.028548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46274 (local address 127.9.77.190:33749)
0810 01:49:45.029056 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:45.029078 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:45.029129 (+ 51us) server_negotiation.cc:407] Connection header received
0810 01:49:45.030130 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:45.030158 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:45.030637 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:45.031072 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:45.031867 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:45.033581 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:45.034512 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:45.035264 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:45.035473 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:45.038581 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:45.038587 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:45.038605 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:45.038640 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:45.041084 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:45.041609 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:45.041624 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:45.041629 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:45.041760 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:45.042101 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:45.042104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:45.042106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:45.042651 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:45.042876 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:45.043204 (+ 328us) server_negotiation.cc:299] Negotiation successful
0810 01:49:45.043610 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":135,"threads_started":1}
I20260810 01:49:45.044962 31066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9bc943de35749e1ac40b04b34a44f6c: No bootstrap required, opened a new log
I20260810 01:49:45.048110 9526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 01:49:45.050230 31066 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9bc943de35749e1ac40b04b34a44f6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bc943de35749e1ac40b04b34a44f6c" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33749 } }
I20260810 01:49:45.050550 31066 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9bc943de35749e1ac40b04b34a44f6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:49:45.050614 31066 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9bc943de35749e1ac40b04b34a44f6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9bc943de35749e1ac40b04b34a44f6c, State: Initialized, Role: FOLLOWER
I20260810 01:49:45.051115 31066 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9bc943de35749e1ac40b04b34a44f6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bc943de35749e1ac40b04b34a44f6c" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33749 } }
I20260810 01:49:45.051256 31066 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9bc943de35749e1ac40b04b34a44f6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:49:45.051302 31066 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9bc943de35749e1ac40b04b34a44f6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:49:45.051383 31066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9bc943de35749e1ac40b04b34a44f6c [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:49:45.052546 31066 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9bc943de35749e1ac40b04b34a44f6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bc943de35749e1ac40b04b34a44f6c" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33749 } }
I20260810 01:49:45.052919 31066 leader_election.cc:302] T 00000000000000000000000000000000 P f9bc943de35749e1ac40b04b34a44f6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9bc943de35749e1ac40b04b34a44f6c; no voters:
I20260810 01:49:45.053313 31066 leader_election.cc:288] T 00000000000000000000000000000000 P f9bc943de35749e1ac40b04b34a44f6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:49:45.053498 31073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9bc943de35749e1ac40b04b34a44f6c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:49:45.053766 31073 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9bc943de35749e1ac40b04b34a44f6c [term 1 LEADER]: Becoming Leader. State: Replica: f9bc943de35749e1ac40b04b34a44f6c, State: Running, Role: LEADER
I20260810 01:49:45.054157 31073 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9bc943de35749e1ac40b04b34a44f6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bc943de35749e1ac40b04b34a44f6c" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33749 } }
I20260810 01:49:45.054857 31066 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9bc943de35749e1ac40b04b34a44f6c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:49:45.056143 31076 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9bc943de35749e1ac40b04b34a44f6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9bc943de35749e1ac40b04b34a44f6c. Latest consensus state: current_term: 1 leader_uuid: "f9bc943de35749e1ac40b04b34a44f6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bc943de35749e1ac40b04b34a44f6c" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33749 } } }
I20260810 01:49:45.056290 31076 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9bc943de35749e1ac40b04b34a44f6c [sys.catalog]: This master's current role is: LEADER
I20260810 01:49:45.056627 31074 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9bc943de35749e1ac40b04b34a44f6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9bc943de35749e1ac40b04b34a44f6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9bc943de35749e1ac40b04b34a44f6c" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 33749 } } }
I20260810 01:49:45.056716 31074 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9bc943de35749e1ac40b04b34a44f6c [sys.catalog]: This master's current role is: LEADER
I20260810 01:49:45.056746 31088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:49:45.060712 31088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:49:45.067596 31088 catalog_manager.cc:1347] Generated new cluster ID: 855195caff04404e914d6670e0e3d26e
I20260810 01:49:45.067691 31088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:49:45.085863 31088 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:49:45.087121 31088 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:49:45.095676 31088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9bc943de35749e1ac40b04b34a44f6c: Generated new TSK 0
I20260810 01:49:45.096839 31088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:49:45.105298 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 30959
2026-08-10T01:49:45Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1769 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:49:45 dist-test-slave-kvfs krb5kdc[31113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:49:45 dist-test-slave-kvfs krb5kdc[31113](info): set up 2 sockets
Aug 10 01:49:45 dist-test-slave-kvfs krb5kdc[31113](info): commencing operation
krb5kdc: starting...
W20260810 01:49:46.410799 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.272s user 0.001s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:49:46 dist-test-slave-kvfs krb5kdc[31113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:49:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:49:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:49:46.563452 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:46111
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:33877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:46111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:49:46.699147 31128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:49:46.699569 31128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:49:46.699654 31128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:49:46.703976 31128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:49:46.704110 31128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:49:46.704147 31128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:49:46.704187 31128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:49:46.704227 31128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:49:46.710160 31128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:33877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:46111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:46111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:49:46.711586 31128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:49:46.712929 31128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:49:46.723681 31134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:46.723693 31135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:46.723827 31128 server_base.cc:1034] running on GCE node
W20260810 01:49:46.723675 31142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:46.724849 31128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:49:46.726320 31128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:49:46.727517 31128 hybrid_clock.cc:648] HybridClock initialized: now 1786326586727494 us; error 39 us; skew 500 ppm
Aug 10 01:49:46 dist-test-slave-kvfs krb5kdc[31113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326586, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:49:46.731210 31128 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:49:46.732520 31128 webserver.cc:460] Webserver started at http://127.9.77.190:39865/ using document root <none> and password file <none>
I20260810 01:49:46.733263 31128 fs_manager.cc:359] Metadata directory not provided
I20260810 01:49:46.733331 31128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:49:46.733614 31128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:49:46.736307 31128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "86545495916040a7803b97b6b4d81c55"
format_stamp: "Formatted at 2026-08-10 01:49:46 on dist-test-slave-kvfs"
server_key: "5afa6600abc93e75b96796e193aef87b"
server_key_iv: "a2292fbc78e7e8af0d4be37ca7052b57"
server_key_version: "encryptionkey@0"
I20260810 01:49:46.737008 31128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "86545495916040a7803b97b6b4d81c55"
format_stamp: "Formatted at 2026-08-10 01:49:46 on dist-test-slave-kvfs"
server_key: "5afa6600abc93e75b96796e193aef87b"
server_key_iv: "a2292fbc78e7e8af0d4be37ca7052b57"
server_key_version: "encryptionkey@0"
I20260810 01:49:46.741707 31128 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 01:49:46.744863 31151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:46.746439 31128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:49:46.746616 31128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "86545495916040a7803b97b6b4d81c55"
format_stamp: "Formatted at 2026-08-10 01:49:46 on dist-test-slave-kvfs"
server_key: "5afa6600abc93e75b96796e193aef87b"
server_key_iv: "a2292fbc78e7e8af0d4be37ca7052b57"
server_key_version: "encryptionkey@0"
I20260810 01:49:46.746762 31128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:49:46.761933 31128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:49:46.766049 31128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:49:46.766296 31128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:49:46.775542 31128 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:46111
I20260810 01:49:46.775918 31228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:46111 every 8 connection(s)
I20260810 01:49:46.776889 31128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:49:46.780529 31229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:49:46.783144 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 31128
I20260810 01:49:46.783313 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:49:46.783707 9526 external_mini_cluster.cc:1428] Setting key 70d04c2a81e3145f934dbccbb984d251
I20260810 01:49:46.789352 31229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55: Bootstrap starting.
I20260810 01:49:46.792790 31229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55: Neither blocks nor log segments found. Creating new log.
I20260810 01:49:46.794011 31229 log.cc:824] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55: Log is configured to *not* fsync() on all Append() calls
Aug 10 01:49:46 dist-test-slave-kvfs krb5kdc[31113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:49:46.797088 31229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55: No bootstrap required, opened a new log
I20260810 01:49:46.801822 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:46.785554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59620 (local address 127.9.77.190:46111)
0810 01:49:46.786097 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:46.786117 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:46.786173 (+ 56us) server_negotiation.cc:407] Connection header received
0810 01:49:46.786977 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:46.787000 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:46.787457 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:46.787820 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:46.788952 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:46.790177 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:46.791505 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:46.792138 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:46.792271 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:46.795758 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:46.795765 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:46.795785 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:46.795819 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:46.797937 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:46.798423 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:46.798430 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:46.798437 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:46.798583 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:46.798903 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:46.798909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:46.798913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:46.799576 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:46.799851 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:46.800215 (+ 364us) server_negotiation.cc:299] Negotiation successful
0810 01:49:46.800602 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":162,"threads_started":1}
I20260810 01:49:46.802469 31229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86545495916040a7803b97b6b4d81c55" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46111 } }
I20260810 01:49:46.802774 31229 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:49:46.802865 31229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86545495916040a7803b97b6b4d81c55, State: Initialized, Role: FOLLOWER
I20260810 01:49:46.803433 31229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86545495916040a7803b97b6b4d81c55" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46111 } }
I20260810 01:49:46.803612 31229 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:49:46.803680 31229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:49:46.803793 31229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:49:46.805056 31229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86545495916040a7803b97b6b4d81c55" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46111 } }
I20260810 01:49:46.805408 31229 leader_election.cc:302] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86545495916040a7803b97b6b4d81c55; no voters:
I20260810 01:49:46.805706 31229 leader_election.cc:288] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:49:46.805922 31234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:49:46.806200 31234 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [term 1 LEADER]: Becoming Leader. State: Replica: 86545495916040a7803b97b6b4d81c55, State: Running, Role: LEADER
I20260810 01:49:46.806624 31234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86545495916040a7803b97b6b4d81c55" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46111 } }
I20260810 01:49:46.807215 31229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:49:46.808648 31235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86545495916040a7803b97b6b4d81c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86545495916040a7803b97b6b4d81c55" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46111 } } }
I20260810 01:49:46.809124 31235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [sys.catalog]: This master's current role is: LEADER
I20260810 01:49:46.808990 31236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86545495916040a7803b97b6b4d81c55. Latest consensus state: current_term: 1 leader_uuid: "86545495916040a7803b97b6b4d81c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86545495916040a7803b97b6b4d81c55" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46111 } } }
I20260810 01:49:46.809619 31254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:49:46.809677 31236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [sys.catalog]: This master's current role is: LEADER
I20260810 01:49:46.813258 31254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:49:46.820367 31254 catalog_manager.cc:1347] Generated new cluster ID: a5b5a3d2f58e40ffaaac013f261a3ac8
I20260810 01:49:46.820456 31254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:49:46.854238 31254 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:49:46.855482 31254 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:49:46.864921 31254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55: Generated new TSK 0
I20260810 01:49:46.866024 31254 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.77.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
I20260810 01:49:46.912238 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:0
--local_ip_for_outbound_sockets=127.9.77.129
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:46111
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:33877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:49:47.052281 31267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:49:47.052651 31267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:49:47.052699 31267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:49:47.056895 31267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:49:47.056962 31267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:49:47.057024 31267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:49:47.062748 31267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:33877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:46111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:49:47.064224 31267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:49:47.065485 31267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:49:47.077280 31276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:47.077312 31277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:47.077345 31280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:47.077661 31267 server_base.cc:1034] running on GCE node
I20260810 01:49:47.078197 31267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:49:47.078980 31267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:49:47.080159 31267 hybrid_clock.cc:648] HybridClock initialized: now 1786326587080123 us; error 57 us; skew 500 ppm
Aug 10 01:49:47 dist-test-slave-kvfs krb5kdc[31113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326587, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:49:47.083510 31267 init.cc:377] Logged in from keytab as kudu/127.9.77.129@KRBTEST.COM (short username kudu)
I20260810 01:49:47.084714 31267 webserver.cc:460] Webserver started at http://127.9.77.129:39747/ using document root <none> and password file <none>
I20260810 01:49:47.085454 31267 fs_manager.cc:359] Metadata directory not provided
I20260810 01:49:47.085557 31267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:49:47.085842 31267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:49:47.088622 31267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/data/instance:
uuid: "1b4562ab42d4461e84c13b245eec5b5a"
format_stamp: "Formatted at 2026-08-10 01:49:47 on dist-test-slave-kvfs"
server_key: "45820d42824fbe0a353bcb7c0d7c4221"
server_key_iv: "79be2319cfc89e2198de4ea0df445122"
server_key_version: "encryptionkey@0"
I20260810 01:49:47.089336 31267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance:
uuid: "1b4562ab42d4461e84c13b245eec5b5a"
format_stamp: "Formatted at 2026-08-10 01:49:47 on dist-test-slave-kvfs"
server_key: "45820d42824fbe0a353bcb7c0d7c4221"
server_key_iv: "79be2319cfc89e2198de4ea0df445122"
server_key_version: "encryptionkey@0"
I20260810 01:49:47.094044 31267 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 01:49:47.097061 31289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:47.098601 31267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 01:49:47.098760 31267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "1b4562ab42d4461e84c13b245eec5b5a"
format_stamp: "Formatted at 2026-08-10 01:49:47 on dist-test-slave-kvfs"
server_key: "45820d42824fbe0a353bcb7c0d7c4221"
server_key_iv: "79be2319cfc89e2198de4ea0df445122"
server_key_version: "encryptionkey@0"
I20260810 01:49:47.098917 31267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:49:47.126367 31267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:49:47.130259 31267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:49:47.130518 31267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:49:47.131304 31267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:49:47.132474 31267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:49:47.132547 31267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:47.132632 31267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:49:47.132681 31267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:47.144371 31267 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:45863
I20260810 01:49:47.144402 31461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:45863 every 8 connection(s)
I20260810 01:49:47.145581 31267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:49:47.152904 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 31267
I20260810 01:49:47.153157 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance
I20260810 01:49:47.153512 9526 external_mini_cluster.cc:1428] Setting key 6fa82768a86594201f11e1562756680b
Aug 10 01:49:47 dist-test-slave-kvfs krb5kdc[31113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326587, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:49:47.162318 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:47.147925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:56787 (local address 127.9.77.190:46111)
0810 01:49:47.148128 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:47.148132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:47.149158 (+ 1026us) server_negotiation.cc:407] Connection header received
0810 01:49:47.150659 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:47.150662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:47.150725 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:47.150807 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:47.152374 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.153087 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.154630 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.154897 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.155019 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:47.158211 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:47.158214 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:47.158216 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:47.158244 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:47.159991 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:47.160587 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:47.160591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:47.160595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:47.160685 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:47.161068 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:47.161071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:47.161073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:47.161288 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:47.161408 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:47.161970 (+ 562us) server_negotiation.cc:299] Negotiation successful
0810 01:49:47.162117 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 01:49:47.163483 31466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:47.148423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.129:56787)
0810 01:49:47.148968 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:47.149019 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:47.150430 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:47.151031 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:47.151038 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:47.151544 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:49:47.152181 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:47.152198 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.153284 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.153287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:47.154495 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:47.154506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.155071 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.155075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:47.155272 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:47.156060 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:49:47.156092 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:49:47.158011 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:49:47.160151 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:47.160157 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:47.160176 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:47.160470 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:47.160781 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:47.160785 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:47.160787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:47.160961 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:47.161398 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:49:47.161406 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:49:47.161705 (+ 299us) client_negotiation.cc:769] Sending connection context
0810 01:49:47.161926 (+ 221us) client_negotiation.cc:240] Negotiation successful
0810 01:49:47.162313 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":129,"threads_started":1}
I20260810 01:49:47.165060 31462 heartbeater.cc:347] Connected to a master server at 127.9.77.190:46111
I20260810 01:49:47.165370 31462 heartbeater.cc:464] Registering TS with master...
I20260810 01:49:47.165990 31462 heartbeater.cc:511] Master 127.9.77.190:46111 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.77.130@KRBTEST.COM; defaulting to no policy
I20260810 01:49:47.168004 31173 ts_manager.cc:194] Registered new tserver with Master: 1b4562ab42d4461e84c13b245eec5b5a (127.9.77.129:45863)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.130@KRBTEST.COM" created.
I20260810 01:49:47.169941 31173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.129@KRBTEST.COM'} at 127.9.77.129:56787
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
I20260810 01:49:47.188529 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:0
--local_ip_for_outbound_sockets=127.9.77.130
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:46111
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:33877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:49:47.334470 31470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:49:47.334874 31470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:49:47.334925 31470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:49:47.339264 31470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:49:47.339336 31470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:49:47.339442 31470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:49:47.345258 31470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:33877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:46111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:49:47.346724 31470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:49:47.348171 31470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 01:49:47.360158 31470 server_base.cc:1034] running on GCE node
W20260810 01:49:47.360185 31478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:47.360179 31475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:47.360363 31476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:47.360978 31470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:49:47.361713 31470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:49:47.362898 31470 hybrid_clock.cc:648] HybridClock initialized: now 1786326587362876 us; error 40 us; skew 500 ppm
Aug 10 01:49:47 dist-test-slave-kvfs krb5kdc[31113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326587, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:49:47.366514 31470 init.cc:377] Logged in from keytab as kudu/127.9.77.130@KRBTEST.COM (short username kudu)
I20260810 01:49:47.367694 31470 webserver.cc:460] Webserver started at http://127.9.77.130:38535/ using document root <none> and password file <none>
I20260810 01:49:47.368496 31470 fs_manager.cc:359] Metadata directory not provided
I20260810 01:49:47.368597 31470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:49:47.368876 31470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:49:47.371474 31470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/data/instance:
uuid: "770f1e22d67044aa9525812b24ddaf97"
format_stamp: "Formatted at 2026-08-10 01:49:47 on dist-test-slave-kvfs"
server_key: "890bde52285b2c23447f6e274877535c"
server_key_iv: "296486718a1f69d42350193f42cd8e31"
server_key_version: "encryptionkey@0"
I20260810 01:49:47.372241 31470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance:
uuid: "770f1e22d67044aa9525812b24ddaf97"
format_stamp: "Formatted at 2026-08-10 01:49:47 on dist-test-slave-kvfs"
server_key: "890bde52285b2c23447f6e274877535c"
server_key_iv: "296486718a1f69d42350193f42cd8e31"
server_key_version: "encryptionkey@0"
I20260810 01:49:47.377090 31470 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 01:49:47.380213 31486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:47.381733 31470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 01:49:47.381888 31470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "770f1e22d67044aa9525812b24ddaf97"
format_stamp: "Formatted at 2026-08-10 01:49:47 on dist-test-slave-kvfs"
server_key: "890bde52285b2c23447f6e274877535c"
server_key_iv: "296486718a1f69d42350193f42cd8e31"
server_key_version: "encryptionkey@0"
I20260810 01:49:47.382030 31470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:49:47.405077 31470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:49:47.409457 31470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:49:47.409698 31470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:49:47.410490 31470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:49:47.411541 31470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:49:47.411616 31470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:47.411696 31470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:49:47.411748 31470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:47.422459 31470 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:40817
I20260810 01:49:47.422484 31654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:40817 every 8 connection(s)
I20260810 01:49:47.423557 31470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:49:47.428747 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 31470
I20260810 01:49:47.428925 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance
I20260810 01:49:47.429291 9526 external_mini_cluster.cc:1428] Setting key a321f478027106096e55440d625d7976
Aug 10 01:49:47 dist-test-slave-kvfs krb5kdc[31113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326587, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:49:47.440727 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:47.426006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:36175 (local address 127.9.77.190:46111)
0810 01:49:47.426221 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:47.426227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:47.427237 (+ 1010us) server_negotiation.cc:407] Connection header received
0810 01:49:47.428793 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:47.428798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:47.428883 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:47.428984 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:47.430574 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.431336 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.432620 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.433016 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.433143 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:47.436391 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:47.436395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:47.436397 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:47.436424 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:47.438028 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:47.438658 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:47.438661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:47.438663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:47.438755 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:47.439199 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:47.439201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:47.439202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:47.439482 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:47.439572 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:47.440427 (+ 855us) server_negotiation.cc:299] Negotiation successful
0810 01:49:47.440562 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 01:49:47.442307 31663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:47.426477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.130:36175)
0810 01:49:47.427050 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:47.427100 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:47.428536 (+ 1436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:47.429225 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:47.429233 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:47.429709 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:49:47.430411 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:47.430432 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.431465 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.431468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:47.432489 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:47.432499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.433196 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.433199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:47.433367 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:47.434164 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:49:47.434192 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:49:47.436195 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:49:47.438178 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:47.438186 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:47.438206 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:47.438513 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:47.438857 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:47.438863 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:47.438866 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:47.439091 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:47.439606 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:49:47.439616 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:49:47.440095 (+ 479us) client_negotiation.cc:769] Sending connection context
0810 01:49:47.440371 (+ 276us) client_negotiation.cc:240] Negotiation successful
0810 01:49:47.440852 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":135,"threads_started":1}
WARNING: no policy specified for kudu/127.9.77.131@KRBTEST.COM; defaulting to no policy
I20260810 01:49:47.443953 31658 heartbeater.cc:347] Connected to a master server at 127.9.77.190:46111
I20260810 01:49:47.444366 31658 heartbeater.cc:464] Registering TS with master...
I20260810 01:49:47.445045 31658 heartbeater.cc:511] Master 127.9.77.190:46111 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.131@KRBTEST.COM" created.
I20260810 01:49:47.446602 31173 ts_manager.cc:194] Registered new tserver with Master: 770f1e22d67044aa9525812b24ddaf97 (127.9.77.130:40817)
I20260810 01:49:47.447700 31173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.130@KRBTEST.COM'} at 127.9.77.130:36175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
I20260810 01:49:47.467896 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:0
--local_ip_for_outbound_sockets=127.9.77.131
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:46111
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:33877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:49:47.603113 31667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:49:47.603492 31667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:49:47.603541 31667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:49:47.608002 31667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:49:47.608131 31667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:49:47.608242 31667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:49:47.614138 31667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:33877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:46111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:49:47.615486 31667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:49:47.616766 31667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:49:47.628189 31674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:47.628189 31680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:47.628191 31676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:47.628559 31667 server_base.cc:1034] running on GCE node
I20260810 01:49:47.629106 31667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:49:47.629947 31667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:49:47.631175 31667 hybrid_clock.cc:648] HybridClock initialized: now 1786326587631141 us; error 43 us; skew 500 ppm
Aug 10 01:49:47 dist-test-slave-kvfs krb5kdc[31113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326587, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:49:47.634845 31667 init.cc:377] Logged in from keytab as kudu/127.9.77.131@KRBTEST.COM (short username kudu)
I20260810 01:49:47.635991 31667 webserver.cc:460] Webserver started at http://127.9.77.131:33231/ using document root <none> and password file <none>
I20260810 01:49:47.636840 31667 fs_manager.cc:359] Metadata directory not provided
I20260810 01:49:47.636953 31667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:49:47.637262 31667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:49:47.640115 31667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/data/instance:
uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e"
format_stamp: "Formatted at 2026-08-10 01:49:47 on dist-test-slave-kvfs"
server_key: "73da1bacad03c3c7f03817b5327b3992"
server_key_iv: "73078766f5a693c62765dde359ad5262"
server_key_version: "encryptionkey@0"
I20260810 01:49:47.640852 31667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance:
uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e"
format_stamp: "Formatted at 2026-08-10 01:49:47 on dist-test-slave-kvfs"
server_key: "73da1bacad03c3c7f03817b5327b3992"
server_key_iv: "73078766f5a693c62765dde359ad5262"
server_key_version: "encryptionkey@0"
I20260810 01:49:47.645859 31667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 01:49:47.649060 31689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:47.650681 31667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 01:49:47.650870 31667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e"
format_stamp: "Formatted at 2026-08-10 01:49:47 on dist-test-slave-kvfs"
server_key: "73da1bacad03c3c7f03817b5327b3992"
server_key_iv: "73078766f5a693c62765dde359ad5262"
server_key_version: "encryptionkey@0"
I20260810 01:49:47.651028 31667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:49:47.682631 31667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:49:47.686848 31667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:49:47.687098 31667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:49:47.687995 31667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:49:47.689146 31667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:49:47.689229 31667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:47.689317 31667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:49:47.689363 31667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:47.699496 31667 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:35807
I20260810 01:49:47.699520 31863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:35807 every 8 connection(s)
I20260810 01:49:47.700668 31667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:49:47.708381 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 31667
I20260810 01:49:47.708550 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance
I20260810 01:49:47.708971 9526 external_mini_cluster.cc:1428] Setting key 59f031868729e9edda123d9f185113b8
Aug 10 01:49:47 dist-test-slave-kvfs krb5kdc[31113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326587, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:49:47.716991 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:47.702881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:55457 (local address 127.9.77.190:46111)
0810 01:49:47.703099 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:47.703104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:47.704218 (+ 1114us) server_negotiation.cc:407] Connection header received
0810 01:49:47.705711 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:47.705714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:47.705772 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:47.705861 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:47.707293 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.708089 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.709698 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.710014 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.710122 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:47.713187 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:47.713189 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:47.713192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:47.713231 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:47.714848 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:47.715322 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:47.715326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:47.715327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:47.715412 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:47.715781 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:47.715784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:47.715785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:47.716076 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:47.716181 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:47.716696 (+ 515us) server_negotiation.cc:299] Negotiation successful
0810 01:49:47.716835 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260810 01:49:47.718056 31866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:47.703440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.131:55457)
0810 01:49:47.703983 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:47.704103 (+ 120us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:47.705489 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:47.706076 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:47.706084 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:47.706496 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:49:47.707135 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:47.707154 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.708239 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.708243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:47.709572 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:47.709585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.710186 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.710191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:47.710398 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:47.711157 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:49:47.711180 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:49:47.713022 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:49:47.714947 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:47.714952 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:47.714967 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:47.715226 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:47.715506 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:47.715509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:47.715510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:47.715651 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:47.716184 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:49:47.716190 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:49:47.716460 (+ 270us) client_negotiation.cc:769] Sending connection context
0810 01:49:47.716639 (+ 179us) client_negotiation.cc:240] Negotiation successful
0810 01:49:47.716943 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":131,"threads_started":1}
I20260810 01:49:47.719453 31864 heartbeater.cc:347] Connected to a master server at 127.9.77.190:46111
I20260810 01:49:47.719717 31864 heartbeater.cc:464] Registering TS with master...
I20260810 01:49:47.720352 31864 heartbeater.cc:511] Master 127.9.77.190:46111 requested a full tablet report, sending...
I20260810 01:49:47.721614 31173 ts_manager.cc:194] Registered new tserver with Master: 4a02a22a13354c07bb0cde6f7c5b4d4e (127.9.77.131:35807)
I20260810 01:49:47.722577 31173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.131@KRBTEST.COM'} at 127.9.77.131:55457
I20260810 01:49:47.723944 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 01:49:47.734558 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:47.726435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59630 (local address 127.9.77.190:46111)
0810 01:49:47.726594 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:47.726600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:47.726820 (+ 220us) server_negotiation.cc:407] Connection header received
0810 01:49:47.726912 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:47.726914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:47.726957 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:47.727025 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:47.727818 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.728451 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.729279 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.729495 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.729574 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:47.731117 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:47.731120 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:47.731123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:47.731148 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:47.732839 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:47.733241 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:47.733244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:47.733245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:47.733327 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:47.733611 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:47.733613 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:47.733614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:47.733880 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:47.733994 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:47.734200 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 01:49:47.734310 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 01:49:47.738566 31173 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 01:49:47.742620 31173 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:49:47.760919 31880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:47.754174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:45863 (local address 127.0.0.1:53996)
0810 01:49:47.755313 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:47.755328 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:47.755496 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:47.756205 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:47.756209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:47.756229 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:49:47.756675 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:47.756683 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.758124 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.758128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:47.759295 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:47.759307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.760252 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.760256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:47.760359 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:47.760413 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 01:49:47.760543 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 01:49:47.760713 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"mutex_wait_us":4,"thread_start_us":97,"threads_started":1}
I20260810 01:49:47.761234 31883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:47.753717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:35807 (local address 127.0.0.1:49088)
0810 01:49:47.755044 (+ 1327us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:47.755058 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:47.755136 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:47.756061 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:47.756066 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:47.756085 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:49:47.756192 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:47.756199 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.757673 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.757677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:47.758853 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:47.758863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.759960 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.759964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:47.760489 (+ 525us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:47.760514 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 01:49:47.760587 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 01:49:47.761083 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":429,"thread_start_us":103,"threads_started":1}
I20260810 01:49:47.761567 31886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:47.754683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53996 (local address 127.9.77.129:45863)
0810 01:49:47.755115 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:47.755124 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:47.755420 (+ 296us) server_negotiation.cc:407] Connection header received
0810 01:49:47.755631 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:47.755636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:47.755837 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:47.756000 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:49:47.756821 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.758003 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.759408 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.760026 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.760157 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:47.760322 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:47.761311 (+ 989us) server_negotiation.cc:299] Negotiation successful
0810 01:49:47.761416 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":118,"threads_started":1}
I20260810 01:49:47.761571 31884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:47.754622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49552 (local address 127.9.77.130:40817)
0810 01:49:47.755158 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:47.755172 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:47.755200 (+ 28us) server_negotiation.cc:407] Connection header received
0810 01:49:47.755321 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:47.755359 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:47.755638 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:47.755836 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:49:47.756232 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.757465 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.759357 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.760532 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.760632 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:47.760768 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:47.761307 (+ 539us) server_negotiation.cc:299] Negotiation successful
0810 01:49:47.761412 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":162,"threads_started":1}
I20260810 01:49:47.761587 31882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:47.753926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49088 (local address 127.9.77.131:35807)
0810 01:49:47.755495 (+ 1569us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:47.755509 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:47.755537 (+ 28us) server_negotiation.cc:407] Connection header received
0810 01:49:47.755629 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:47.755634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:47.755837 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:47.756000 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:49:47.756305 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.757512 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.758987 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.759819 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.759938 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:47.760162 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:47.760685 (+ 523us) server_negotiation.cc:299] Negotiation successful
0810 01:49:47.760823 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1436,"thread_start_us":147,"threads_started":1}
I20260810 01:49:47.761811 31881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:47.753938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:40817 (local address 127.0.0.1:49552)
0810 01:49:47.754885 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:47.754949 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:47.755114 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:47.755836 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:47.755843 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:47.755867 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:49:47.756116 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:47.756124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.757622 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.757627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:47.759212 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:47.759223 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.760840 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.760844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:47.760950 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:47.761014 (+ 64us) client_negotiation.cc:769] Sending connection context
0810 01:49:47.761091 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 01:49:47.761165 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":131,"thread_start_us":145,"threads_started":1}
I20260810 01:49:47.765060 31767 tablet_service.cc:1467] Processing CreateTablet for tablet f00e278d4c2b46fea4c799e472d4ce63 (DEFAULT_TABLE table=test-table [id=1f56139e917544c0b845d01e13fc83ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:49:47.765056 31362 tablet_service.cc:1467] Processing CreateTablet for tablet f00e278d4c2b46fea4c799e472d4ce63 (DEFAULT_TABLE table=test-table [id=1f56139e917544c0b845d01e13fc83ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:49:47.766322 31767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f00e278d4c2b46fea4c799e472d4ce63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:49:47.766324 31362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f00e278d4c2b46fea4c799e472d4ce63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:49:47.766162 31559 tablet_service.cc:1467] Processing CreateTablet for tablet f00e278d4c2b46fea4c799e472d4ce63 (DEFAULT_TABLE table=test-table [id=1f56139e917544c0b845d01e13fc83ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:49:47.767206 31559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f00e278d4c2b46fea4c799e472d4ce63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:49:47.774099 31887 tablet_bootstrap.cc:492] T f00e278d4c2b46fea4c799e472d4ce63 P 4a02a22a13354c07bb0cde6f7c5b4d4e: Bootstrap starting.
I20260810 01:49:47.775446 31889 tablet_bootstrap.cc:492] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97: Bootstrap starting.
I20260810 01:49:47.776214 31888 tablet_bootstrap.cc:492] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a: Bootstrap starting.
I20260810 01:49:47.777853 31887 tablet_bootstrap.cc:654] T f00e278d4c2b46fea4c799e472d4ce63 P 4a02a22a13354c07bb0cde6f7c5b4d4e: Neither blocks nor log segments found. Creating new log.
I20260810 01:49:47.778322 31889 tablet_bootstrap.cc:654] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97: Neither blocks nor log segments found. Creating new log.
I20260810 01:49:47.778453 31888 tablet_bootstrap.cc:654] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a: Neither blocks nor log segments found. Creating new log.
I20260810 01:49:47.779386 31889 log.cc:824] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97: Log is configured to *not* fsync() on all Append() calls
I20260810 01:49:47.779386 31888 log.cc:824] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a: Log is configured to *not* fsync() on all Append() calls
I20260810 01:49:47.779385 31887 log.cc:824] T f00e278d4c2b46fea4c799e472d4ce63 P 4a02a22a13354c07bb0cde6f7c5b4d4e: Log is configured to *not* fsync() on all Append() calls
I20260810 01:49:47.782022 31888 tablet_bootstrap.cc:492] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a: No bootstrap required, opened a new log
I20260810 01:49:47.782022 31889 tablet_bootstrap.cc:492] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97: No bootstrap required, opened a new log
I20260810 01:49:47.782261 31888 ts_tablet_manager.cc:1397] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 01:49:47.782326 31887 tablet_bootstrap.cc:492] T f00e278d4c2b46fea4c799e472d4ce63 P 4a02a22a13354c07bb0cde6f7c5b4d4e: No bootstrap required, opened a new log
I20260810 01:49:47.782626 31887 ts_tablet_manager.cc:1397] T f00e278d4c2b46fea4c799e472d4ce63 P 4a02a22a13354c07bb0cde6f7c5b4d4e: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260810 01:49:47.783139 31889 ts_tablet_manager.cc:1397] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 01:49:47.787048 31887 raft_consensus.cc:348] T f00e278d4c2b46fea4c799e472d4ce63 P 4a02a22a13354c07bb0cde6f7c5b4d4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } }
I20260810 01:49:47.787576 31887 raft_consensus.cc:374] T f00e278d4c2b46fea4c799e472d4ce63 P 4a02a22a13354c07bb0cde6f7c5b4d4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:49:47.787475 31889 raft_consensus.cc:348] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } }
I20260810 01:49:47.787475 31888 raft_consensus.cc:348] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } }
I20260810 01:49:47.787755 31888 raft_consensus.cc:374] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:49:47.787688 31887 raft_consensus.cc:729] T f00e278d4c2b46fea4c799e472d4ce63 P 4a02a22a13354c07bb0cde6f7c5b4d4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a02a22a13354c07bb0cde6f7c5b4d4e, State: Initialized, Role: FOLLOWER
I20260810 01:49:47.787755 31889 raft_consensus.cc:374] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:49:47.787825 31888 raft_consensus.cc:729] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4562ab42d4461e84c13b245eec5b5a, State: Initialized, Role: FOLLOWER
I20260810 01:49:47.787825 31889 raft_consensus.cc:729] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770f1e22d67044aa9525812b24ddaf97, State: Initialized, Role: FOLLOWER
I20260810 01:49:47.788461 31888 consensus_queue.cc:260] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } }
I20260810 01:49:47.788462 31889 consensus_queue.cc:260] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } }
I20260810 01:49:47.788463 31887 consensus_queue.cc:260] T f00e278d4c2b46fea4c799e472d4ce63 P 4a02a22a13354c07bb0cde6f7c5b4d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } }
I20260810 01:49:47.789741 31887 ts_tablet_manager.cc:1428] T f00e278d4c2b46fea4c799e472d4ce63 P 4a02a22a13354c07bb0cde6f7c5b4d4e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 01:49:47.789743 31888 ts_tablet_manager.cc:1428] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 01:49:47.789743 31889 ts_tablet_manager.cc:1428] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 01:49:47.790203 31658 heartbeater.cc:503] Master 127.9.77.190:46111 was elected leader, sending a full tablet report...
I20260810 01:49:47.790199 31462 heartbeater.cc:503] Master 127.9.77.190:46111 was elected leader, sending a full tablet report...
I20260810 01:49:47.793237 31864 heartbeater.cc:503] Master 127.9.77.190:46111 was elected leader, sending a full tablet report...
W20260810 01:49:47.897423 31464 tablet.cc:2369] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 01:49:47.925065 31661 tablet.cc:2369] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 01:49:47.951817 31865 tablet.cc:2369] T f00e278d4c2b46fea4c799e472d4ce63 P 4a02a22a13354c07bb0cde6f7c5b4d4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 01:49:47.965198 31900 raft_consensus.cc:482] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:49:47.965533 31900 raft_consensus.cc:504] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } }
I20260810 01:49:47.967111 31900 leader_election.cc:288] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a02a22a13354c07bb0cde6f7c5b4d4e (127.9.77.131:35807), 1b4562ab42d4461e84c13b245eec5b5a (127.9.77.129:45863)
I20260810 01:49:47.971628 31886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:47.967659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:45549 (local address 127.9.77.129:45863)
0810 01:49:47.967813 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:47.967818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:47.968076 (+ 258us) server_negotiation.cc:407] Connection header received
0810 01:49:47.968238 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:47.968241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:47.968305 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:47.968398 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:49:47.968731 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.969298 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.970520 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.970984 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.971120 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:47.971180 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:47.971400 (+ 220us) server_negotiation.cc:299] Negotiation successful
0810 01:49:47.971492 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 01:49:47.971628 31882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:47.967565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:35769 (local address 127.9.77.131:35807)
0810 01:49:47.967755 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:47.967760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:47.967989 (+ 229us) server_negotiation.cc:407] Connection header received
0810 01:49:47.968235 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:47.968238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:47.968323 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:47.968423 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:49:47.968739 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.969291 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.970518 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.970962 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.971100 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:47.971171 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:47.971380 (+ 209us) server_negotiation.cc:299] Negotiation successful
0810 01:49:47.971470 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 01:49:47.971645 31904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:47.967419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:35807 (local address 127.9.77.130:35769)
0810 01:49:47.967872 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:47.967886 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:47.968076 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:47.968446 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:47.968449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:47.968494 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:49:47.968627 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:47.968632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.969407 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.969411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:47.970384 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:47.970393 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.971097 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.971101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:47.971206 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:47.971255 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 01:49:47.971352 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 01:49:47.971441 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":69,"spinlock_wait_cycles":768,"thread_start_us":174,"threads_started":1}
I20260810 01:49:47.971652 31903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:47.967436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:45863 (local address 127.9.77.130:45549)
0810 01:49:47.967948 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:47.967960 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:47.968128 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:47.968415 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:47.968419 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:47.968435 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:49:47.968612 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:47.968619 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.969423 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.969427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:47.970380 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:47.970390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.971112 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.971116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:47.971232 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:47.971288 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 01:49:47.971379 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 01:49:47.971456 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":47,"spinlock_wait_cycles":14080,"thread_start_us":119,"threads_started":1}
I20260810 01:49:47.972571 31392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f00e278d4c2b46fea4c799e472d4ce63" candidate_uuid: "770f1e22d67044aa9525812b24ddaf97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4562ab42d4461e84c13b245eec5b5a" is_pre_election: true
I20260810 01:49:47.972553 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f00e278d4c2b46fea4c799e472d4ce63" candidate_uuid: "770f1e22d67044aa9525812b24ddaf97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" is_pre_election: true
I20260810 01:49:47.972998 31795 raft_consensus.cc:2454] T f00e278d4c2b46fea4c799e472d4ce63 P 4a02a22a13354c07bb0cde6f7c5b4d4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 770f1e22d67044aa9525812b24ddaf97 in term 0.
I20260810 01:49:47.972998 31392 raft_consensus.cc:2454] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 770f1e22d67044aa9525812b24ddaf97 in term 0.
I20260810 01:49:47.973640 31487 leader_election.cc:302] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1b4562ab42d4461e84c13b245eec5b5a, 4a02a22a13354c07bb0cde6f7c5b4d4e, 770f1e22d67044aa9525812b24ddaf97; no voters:
I20260810 01:49:47.974000 31900 raft_consensus.cc:2781] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:49:47.974100 31900 raft_consensus.cc:482] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:49:47.974144 31900 raft_consensus.cc:3037] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:49:47.975517 31900 raft_consensus.cc:504] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } }
I20260810 01:49:47.975968 31900 leader_election.cc:288] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97 [CANDIDATE]: Term 1 election: Requested vote from peers 4a02a22a13354c07bb0cde6f7c5b4d4e (127.9.77.131:35807), 1b4562ab42d4461e84c13b245eec5b5a (127.9.77.129:45863)
I20260810 01:49:47.976439 31795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f00e278d4c2b46fea4c799e472d4ce63" candidate_uuid: "770f1e22d67044aa9525812b24ddaf97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e"
I20260810 01:49:47.976498 31392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f00e278d4c2b46fea4c799e472d4ce63" candidate_uuid: "770f1e22d67044aa9525812b24ddaf97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4562ab42d4461e84c13b245eec5b5a"
I20260810 01:49:47.976604 31795 raft_consensus.cc:3037] T f00e278d4c2b46fea4c799e472d4ce63 P 4a02a22a13354c07bb0cde6f7c5b4d4e [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:49:47.976636 31392 raft_consensus.cc:3037] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:49:47.978199 31392 raft_consensus.cc:2454] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 770f1e22d67044aa9525812b24ddaf97 in term 1.
I20260810 01:49:47.978199 31795 raft_consensus.cc:2454] T f00e278d4c2b46fea4c799e472d4ce63 P 4a02a22a13354c07bb0cde6f7c5b4d4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 770f1e22d67044aa9525812b24ddaf97 in term 1.
I20260810 01:49:47.978571 31489 leader_election.cc:302] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4562ab42d4461e84c13b245eec5b5a, 770f1e22d67044aa9525812b24ddaf97; no voters:
I20260810 01:49:47.978775 31900 raft_consensus.cc:2781] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:49:47.979110 31900 raft_consensus.cc:686] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97 [term 1 LEADER]: Becoming Leader. State: Replica: 770f1e22d67044aa9525812b24ddaf97, State: Running, Role: LEADER
I20260810 01:49:47.979552 31900 consensus_queue.cc:237] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } }
I20260810 01:49:47.983598 31168 catalog_manager.cc:5495] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 770f1e22d67044aa9525812b24ddaf97 (127.9.77.130). New cstate: current_term: 1 leader_uuid: "770f1e22d67044aa9525812b24ddaf97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } health_report { overall_health: UNKNOWN } } }
I20260810 01:49:47.999193 31884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:47.995012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49558 (local address 127.9.77.130:40817)
0810 01:49:47.995166 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:47.995170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:47.995190 (+ 20us) server_negotiation.cc:407] Connection header received
0810 01:49:47.995294 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:47.995296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:47.995365 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:47.995480 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:49:47.995713 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.996418 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.997370 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:47.997625 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:47.997728 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:47.998099 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:49:47.998704 (+ 605us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:49:47.998822 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:47.998966 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 01:49:47.999041 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 01:49:48.007439 31392 raft_consensus.cc:1261] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a [term 1 FOLLOWER]: Refusing update from remote peer 770f1e22d67044aa9525812b24ddaf97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:49:48.007439 31795 raft_consensus.cc:1261] T f00e278d4c2b46fea4c799e472d4ce63 P 4a02a22a13354c07bb0cde6f7c5b4d4e [term 1 FOLLOWER]: Refusing update from remote peer 770f1e22d67044aa9525812b24ddaf97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:49:48.008509 31900 consensus_queue.cc:1035] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:49:48.008812 31906 consensus_queue.cc:1035] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:49:48.018857 31914 mvcc.cc:205] Tried to move back new op lower bound from 7316793704467976192 to 7316793704367910912. Current Snapshot: MvccSnapshot[applied={T|T < 7316793704467976192}]
I20260810 01:49:48.023267 31915 mvcc.cc:205] Tried to move back new op lower bound from 7316793704467976192 to 7316793704367910912. Current Snapshot: MvccSnapshot[applied={T|T < 7316793704467976192 or (T in {7316793704467976192})}]
I20260810 01:49:48.030653 31168 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:49:48.030937 31168 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:49:48.034411 31168 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55: Sending DeleteTablet for 3 replicas of tablet f00e278d4c2b46fea4c799e472d4ce63
I20260810 01:49:48.035585 31559 tablet_service.cc:1514] Processing DeleteTablet for tablet f00e278d4c2b46fea4c799e472d4ce63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:49:48 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:49552
I20260810 01:49:48.035760 31362 tablet_service.cc:1514] Processing DeleteTablet for tablet f00e278d4c2b46fea4c799e472d4ce63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:49:48 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:53996
I20260810 01:49:48.035764 31767 tablet_service.cc:1514] Processing DeleteTablet for tablet f00e278d4c2b46fea4c799e472d4ce63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:49:48 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:49088
I20260810 01:49:48.036428 31930 tablet_replica.cc:330] T f00e278d4c2b46fea4c799e472d4ce63 P 4a02a22a13354c07bb0cde6f7c5b4d4e: stopping tablet replica
I20260810 01:49:48.036566 31929 tablet_replica.cc:330] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97: stopping tablet replica
I20260810 01:49:48.036674 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 31267
I20260810 01:49:48.040532 31930 raft_consensus.cc:2229] T f00e278d4c2b46fea4c799e472d4ce63 P 4a02a22a13354c07bb0cde6f7c5b4d4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:49:48.041009 31930 raft_consensus.cc:2258] T f00e278d4c2b46fea4c799e472d4ce63 P 4a02a22a13354c07bb0cde6f7c5b4d4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:49:48.045795 31930 ts_tablet_manager.cc:1905] T f00e278d4c2b46fea4c799e472d4ce63 P 4a02a22a13354c07bb0cde6f7c5b4d4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:49:48.049841 31930 ts_tablet_manager.cc:1918] T f00e278d4c2b46fea4c799e472d4ce63 P 4a02a22a13354c07bb0cde6f7c5b4d4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:49:48.049971 31930 log.cc:1196] T f00e278d4c2b46fea4c799e472d4ce63 P 4a02a22a13354c07bb0cde6f7c5b4d4e: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/wal/wals/f00e278d4c2b46fea4c799e472d4ce63
I20260810 01:49:48.050453 31930 ts_tablet_manager.cc:1939] T f00e278d4c2b46fea4c799e472d4ce63 P 4a02a22a13354c07bb0cde6f7c5b4d4e: Deleting consensus metadata
I20260810 01:49:48.055017 31152 catalog_manager.cc:4841] TS 4a02a22a13354c07bb0cde6f7c5b4d4e (127.9.77.131:35807): tablet f00e278d4c2b46fea4c799e472d4ce63 (table test-table [id=1f56139e917544c0b845d01e13fc83ab]) successfully deleted
I20260810 01:49:48.063831 31929 raft_consensus.cc:2229] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97 [term 1 LEADER]: Raft consensus shutting down.
W20260810 01:49:48.063897 31153 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.77.129:45863 (error 108)
I20260810 01:49:48.064461 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 31470
I20260810 01:49:48.076432 31883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:48.068197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:45863 (local address 127.0.0.1:54004)
0810 01:49:48.076090 (+ 7893us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:48.076244 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.129:45863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7692}
W20260810 01:49:48.077792 31153 catalog_manager.cc:4584] TS 1b4562ab42d4461e84c13b245eec5b5a (127.9.77.129:45863): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f00e278d4c2b46fea4c799e472d4ce63: Network error: Client connection negotiation failed: client connection to 127.9.77.129:45863: connect: Connection refused (error 111)
I20260810 01:49:48.085418 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 31667
I20260810 01:49:48.093251 31883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:48.085731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:40817 (local address 127.0.0.1:49572)
0810 01:49:48.092961 (+ 7230us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:48.093077 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.130:40817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7050}
I20260810 01:49:48.101876 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 31128
I20260810 01:49:48.116593 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:46111
--webserver_interface=127.9.77.190
--webserver_port=39865
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:33877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:46111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:49:48.251778 31932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:49:48.252218 31932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:49:48.252271 31932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:49:48.256501 31932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:49:48.256574 31932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:49:48.256594 31932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:49:48.256611 31932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:49:48.256628 31932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:49:48.262518 31932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:33877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:46111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:46111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=39865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:49:48.263904 31932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:49:48.265285 31932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:49:48.284564 31939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:48.284564 31938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:48.284566 31942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:48.285324 31932 server_base.cc:1034] running on GCE node
I20260810 01:49:48.286226 31932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:49:48.287864 31932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:49:48.289117 31932 hybrid_clock.cc:648] HybridClock initialized: now 1786326588289078 us; error 54 us; skew 500 ppm
Aug 10 01:49:48 dist-test-slave-kvfs krb5kdc[31113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326588, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:49:48.292780 31932 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:49:48.293973 31932 webserver.cc:460] Webserver started at http://127.9.77.190:39865/ using document root <none> and password file <none>
I20260810 01:49:48.294731 31932 fs_manager.cc:359] Metadata directory not provided
I20260810 01:49:48.294859 31932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:49:48.299733 31932 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 01:49:48.302472 31949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:48.304221 31932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 01:49:48.304391 31932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "86545495916040a7803b97b6b4d81c55"
format_stamp: "Formatted at 2026-08-10 01:49:46 on dist-test-slave-kvfs"
server_key: "5afa6600abc93e75b96796e193aef87b"
server_key_iv: "a2292fbc78e7e8af0d4be37ca7052b57"
server_key_version: "encryptionkey@0"
I20260810 01:49:48.305179 31932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:49:48.332299 31932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:49:48.336721 31932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:49:48.336999 31932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:49:48.345947 31932 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:46111
I20260810 01:49:48.346419 32033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:46111 every 8 connection(s)
I20260810 01:49:48.347539 31932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:49:48.353124 32034 sys_catalog.cc:263] Verifying existing consensus state
I20260810 01:49:48.355106 32034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55: Bootstrap starting.
I20260810 01:49:48.356585 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 31932
I20260810 01:49:48.357149 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:45863
--local_ip_for_outbound_sockets=127.9.77.129
--tserver_master_addrs=127.9.77.190:46111
--webserver_port=39747
--webserver_interface=127.9.77.129
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:33877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 01:49:48.381361 32034 log.cc:824] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55: Log is configured to *not* fsync() on all Append() calls
I20260810 01:49:48.393065 32034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 01:49:48.393569 32034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55: Bootstrap complete.
I20260810 01:49:48.398635 32034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86545495916040a7803b97b6b4d81c55" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46111 } }
I20260810 01:49:48.399170 32034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86545495916040a7803b97b6b4d81c55, State: Initialized, Role: FOLLOWER
I20260810 01:49:48.399832 32034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86545495916040a7803b97b6b4d81c55" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46111 } }
I20260810 01:49:48.400051 32034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:49:48.400171 32034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:49:48.400311 32034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [term 1 FOLLOWER]: Advancing to term 2
I20260810 01:49:48.402757 32034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86545495916040a7803b97b6b4d81c55" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46111 } }
I20260810 01:49:48.403203 32034 leader_election.cc:302] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86545495916040a7803b97b6b4d81c55; no voters:
I20260810 01:49:48.403611 32034 leader_election.cc:288] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 01:49:48.403770 32038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 01:49:48.404080 32038 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [term 2 LEADER]: Becoming Leader. State: Replica: 86545495916040a7803b97b6b4d81c55, State: Running, Role: LEADER
I20260810 01:49:48.404462 32038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86545495916040a7803b97b6b4d81c55" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46111 } }
I20260810 01:49:48.405264 32034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:49:48.406271 32040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86545495916040a7803b97b6b4d81c55. Latest consensus state: current_term: 2 leader_uuid: "86545495916040a7803b97b6b4d81c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86545495916040a7803b97b6b4d81c55" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46111 } } }
I20260810 01:49:48.406462 32040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [sys.catalog]: This master's current role is: LEADER
I20260810 01:49:48.406307 32039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "86545495916040a7803b97b6b4d81c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86545495916040a7803b97b6b4d81c55" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46111 } } }
I20260810 01:49:48.406771 32039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [sys.catalog]: This master's current role is: LEADER
I20260810 01:49:48.406898 32046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:49:48.412891 32046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:49:48.413578 32046 catalog_manager.cc:1259] Loaded cluster ID: a5b5a3d2f58e40ffaaac013f261a3ac8
I20260810 01:49:48.413693 32046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:49:48.415292 32046 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 01:49:48.419668 32046 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 01:49:48.426507 32046 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:49:48.427382 32046 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55: Loaded TSK: 0
I20260810 01:49:48.428843 32046 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 01:49:48.509888 32036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:49:48.510193 32036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:49:48.510238 32036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:49:48.514411 32036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:49:48.514477 32036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:49:48.514539 32036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:49:48.520136 32036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:33877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:45863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=39747
--tserver_master_addrs=127.9.77.190:46111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:49:48.521437 32036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:49:48.522725 32036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:49:48.535506 32072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:48.535662 32075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:48.535962 32036 server_base.cc:1034] running on GCE node
W20260810 01:49:48.535485 32070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:48.536603 32036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:49:48.537413 32036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:49:48.538618 32036 hybrid_clock.cc:648] HybridClock initialized: now 1786326588538580 us; error 57 us; skew 500 ppm
Aug 10 01:49:48 dist-test-slave-kvfs krb5kdc[31113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326588, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:49:48.541991 32036 init.cc:377] Logged in from keytab as kudu/127.9.77.129@KRBTEST.COM (short username kudu)
I20260810 01:49:48.543097 32036 webserver.cc:460] Webserver started at http://127.9.77.129:39747/ using document root <none> and password file <none>
I20260810 01:49:48.543798 32036 fs_manager.cc:359] Metadata directory not provided
I20260810 01:49:48.543881 32036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:49:48.549346 32036 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 01:49:48.552480 32083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:48.554332 32036 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 01:49:48.554524 32036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "1b4562ab42d4461e84c13b245eec5b5a"
format_stamp: "Formatted at 2026-08-10 01:49:47 on dist-test-slave-kvfs"
server_key: "45820d42824fbe0a353bcb7c0d7c4221"
server_key_iv: "79be2319cfc89e2198de4ea0df445122"
server_key_version: "encryptionkey@0"
I20260810 01:49:48.555326 32036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:49:48.585564 32036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:49:48.589977 32036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:49:48.590260 32036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:49:48.591130 32036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:49:48.592621 32095 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 01:49:48.597903 32036 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 01:49:48.597997 32036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260810 01:49:48.598102 32036 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 01:49:48.600320 32036 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 01:49:48.600392 32036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260810 01:49:48.600812 32095 tablet_bootstrap.cc:492] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a: Bootstrap starting.
I20260810 01:49:48.612936 32036 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:45863
I20260810 01:49:48.613039 32261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:45863 every 8 connection(s)
I20260810 01:49:48.614266 32036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:49:48.619434 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 32036
I20260810 01:49:48.619985 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:40817
--local_ip_for_outbound_sockets=127.9.77.130
--tserver_master_addrs=127.9.77.190:46111
--webserver_port=38535
--webserver_interface=127.9.77.130
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:33877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 01:49:48.623898 32095 log.cc:824] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a: Log is configured to *not* fsync() on all Append() calls
Aug 10 01:49:48 dist-test-slave-kvfs krb5kdc[31113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326588, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:49:48.634784 32095 tablet_bootstrap.cc:492] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 01:49:48.635371 32095 tablet_bootstrap.cc:492] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a: Bootstrap complete.
I20260810 01:49:48.636947 32095 ts_tablet_manager.cc:1397] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a: Time spent bootstrapping tablet: real 0.036s user 0.027s sys 0.008s
I20260810 01:49:48.638170 32269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:48.617201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.129:42781)
0810 01:49:48.617899 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:48.617953 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:48.618983 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:48.620858 (+ 1875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:48.620871 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:48.621507 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:49:48.622321 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:48.622350 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:48.624567 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:48.624572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:48.626085 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:48.626099 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:48.627283 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:48.627290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:48.627536 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:48.628457 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:49:48.628480 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:49:48.630877 (+ 2397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:49:48.633824 (+ 2947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:48.633835 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:48.633857 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:48.634310 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:48.634800 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:48.634806 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:48.634810 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:48.635029 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:48.635915 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:49:48.635923 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:49:48.636346 (+ 423us) client_negotiation.cc:769] Sending connection context
0810 01:49:48.636605 (+ 259us) client_negotiation.cc:240] Negotiation successful
0810 01:49:48.637024 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":238,"threads_started":1}
I20260810 01:49:48.638171 32268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:48.617183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:42781 (local address 127.9.77.190:46111)
0810 01:49:48.617653 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:48.617678 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:48.618104 (+ 426us) server_negotiation.cc:407] Connection header received
0810 01:49:48.619726 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:48.619744 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:48.620312 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:48.620799 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:48.622577 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:48.624348 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:48.626257 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:48.627089 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:48.627339 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:48.631157 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:48.631167 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:48.631187 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:48.631232 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:48.633666 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:48.634461 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:48.634472 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:48.634479 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:48.634662 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:48.635154 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:48.635161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:48.635167 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:48.635755 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:48.635897 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:48.636757 (+ 860us) server_negotiation.cc:299] Negotiation successful
0810 01:49:48.637131 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":163,"threads_started":1}
I20260810 01:49:48.640318 32095 raft_consensus.cc:348] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } }
I20260810 01:49:48.640950 32095 raft_consensus.cc:729] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4562ab42d4461e84c13b245eec5b5a, State: Initialized, Role: FOLLOWER
I20260810 01:49:48.641599 32095 consensus_queue.cc:260] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } }
I20260810 01:49:48.641944 32262 heartbeater.cc:347] Connected to a master server at 127.9.77.190:46111
I20260810 01:49:48.642230 32262 heartbeater.cc:464] Registering TS with master...
I20260810 01:49:48.642892 32262 heartbeater.cc:511] Master 127.9.77.190:46111 requested a full tablet report, sending...
I20260810 01:49:48.642904 32095 ts_tablet_manager.cc:1428] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 01:49:48.646108 31976 ts_manager.cc:194] Registered new tserver with Master: 1b4562ab42d4461e84c13b245eec5b5a (127.9.77.129:45863)
I20260810 01:49:48.650576 31976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.129@KRBTEST.COM'} at 127.9.77.129:42781
I20260810 01:49:48.653837 32262 heartbeater.cc:503] Master 127.9.77.190:46111 was elected leader, sending a full tablet report...
Aug 10 01:49:48 dist-test-slave-kvfs krb5kdc[31113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326588, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for kudu/127.9.77.129@KRBTEST.COM
I20260810 01:49:48.662858 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:48.649865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:45863 (local address 127.0.0.1:54012)
0810 01:49:48.650775 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:48.650830 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:48.651002 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:48.651747 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:48.651753 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:48.652302 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:49:48.652462 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:48.652471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:48.653880 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:48.653884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:48.654884 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:48.654896 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:48.655743 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:48.655746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:48.655872 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:48.656582 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:49:48.656599 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:49:48.658567 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:49:48.660764 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:48.660774 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:48.660780 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:48.661096 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:48.661393 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:48.661397 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:48.661399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:48.661502 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:48.662112 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:49:48.662118 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:49:48.662279 (+ 161us) client_negotiation.cc:769] Sending connection context
0810 01:49:48.662440 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 01:49:48.662668 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":146,"threads_started":1}
I20260810 01:49:48.662868 32276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:48.649979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54012 (local address 127.9.77.129:45863)
0810 01:49:48.650469 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:48.650482 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:48.650955 (+ 473us) server_negotiation.cc:407] Connection header received
0810 01:49:48.651180 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:48.651186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:48.651424 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:48.651593 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:48.652612 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:48.653741 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:48.655026 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:48.655571 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:48.655680 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:48.658714 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:48.658724 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:48.658737 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:48.658763 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:48.660656 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:48.661206 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:48.661210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:48.661212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:48.661298 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:48.661665 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:48.661668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:48.661670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:48.661986 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:48.662109 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:48.662523 (+ 414us) server_negotiation.cc:299] Negotiation successful
0810 01:49:48.662691 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":376,"thread_start_us":130,"threads_started":1}
I20260810 01:49:48.664324 32163 tablet_service.cc:1514] Processing DeleteTablet for tablet f00e278d4c2b46fea4c799e472d4ce63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:49:48 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:54012
I20260810 01:49:48.665498 32278 tablet_replica.cc:330] stopping tablet replica
I20260810 01:49:48.665899 32278 raft_consensus.cc:2229] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:49:48.666229 32278 raft_consensus.cc:2258] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:49:48.667665 32278 ts_tablet_manager.cc:1905] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:49:48.673133 32278 ts_tablet_manager.cc:1918] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:49:48.673276 32278 log.cc:1196] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/wal/wals/f00e278d4c2b46fea4c799e472d4ce63
I20260810 01:49:48.673607 32278 ts_tablet_manager.cc:1939] T f00e278d4c2b46fea4c799e472d4ce63 P 1b4562ab42d4461e84c13b245eec5b5a: Deleting consensus metadata
I20260810 01:49:48.676080 31953 catalog_manager.cc:4841] TS 1b4562ab42d4461e84c13b245eec5b5a (127.9.77.129:45863): tablet f00e278d4c2b46fea4c799e472d4ce63 (table test-table [id=1f56139e917544c0b845d01e13fc83ab]) successfully deleted
W20260810 01:49:48.769505 32270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:49:48.769896 32270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:49:48.769946 32270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:49:48.774293 32270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:49:48.774384 32270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:49:48.774451 32270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:49:48.780210 32270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:33877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:40817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=38535
--tserver_master_addrs=127.9.77.190:46111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:49:48.781868 32270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:49:48.783231 32270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:49:48.795089 32285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:48.795117 32286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:48.795312 32270 server_base.cc:1034] running on GCE node
W20260810 01:49:48.795179 32289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:48.796003 32270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:49:48.796895 32270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:49:48.798127 32270 hybrid_clock.cc:648] HybridClock initialized: now 1786326588798095 us; error 51 us; skew 500 ppm
Aug 10 01:49:48 dist-test-slave-kvfs krb5kdc[31113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326588, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:49:48.805750 32270 init.cc:377] Logged in from keytab as kudu/127.9.77.130@KRBTEST.COM (short username kudu)
I20260810 01:49:48.807063 32270 webserver.cc:460] Webserver started at http://127.9.77.130:38535/ using document root <none> and password file <none>
I20260810 01:49:48.807842 32270 fs_manager.cc:359] Metadata directory not provided
I20260810 01:49:48.807961 32270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:49:48.812978 32270 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 01:49:48.815647 32299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:48.817314 32270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 01:49:48.817476 32270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "770f1e22d67044aa9525812b24ddaf97"
format_stamp: "Formatted at 2026-08-10 01:49:47 on dist-test-slave-kvfs"
server_key: "890bde52285b2c23447f6e274877535c"
server_key_iv: "296486718a1f69d42350193f42cd8e31"
server_key_version: "encryptionkey@0"
I20260810 01:49:48.818166 32270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:49:48.844944 32270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:49:48.849140 32270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:49:48.849431 32270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:49:48.850286 32270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:49:48.851691 32308 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 01:49:48.857239 32270 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 01:49:48.857360 32270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260810 01:49:48.857479 32270 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 01:49:48.859702 32270 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 01:49:48.859778 32270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 01:49:48.860247 32308 tablet_bootstrap.cc:492] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97: Bootstrap starting.
I20260810 01:49:48.872802 32270 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:40817
I20260810 01:49:48.872871 32472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:40817 every 8 connection(s)
I20260810 01:49:48.874085 32270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:49:48.882629 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 32270
I20260810 01:49:48.883255 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:35807
--local_ip_for_outbound_sockets=127.9.77.131
--tserver_master_addrs=127.9.77.190:46111
--webserver_port=33231
--webserver_interface=127.9.77.131
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:33877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 01:49:48.884317 32308 log.cc:824] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97: Log is configured to *not* fsync() on all Append() calls
Aug 10 01:49:48 dist-test-slave-kvfs krb5kdc[31113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326588, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:49:48.893312 32268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:48.876889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:45535 (local address 127.9.77.190:46111)
0810 01:49:48.877102 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:48.877108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:48.878022 (+ 914us) server_negotiation.cc:407] Connection header received
0810 01:49:48.879584 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:48.879587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:48.879659 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:48.879734 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:48.881909 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:48.882551 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:48.884220 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:48.884542 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:48.884679 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:48.888324 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:48.888328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:48.888331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:48.888369 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:48.890328 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:48.891021 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:48.891027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:48.891032 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:48.891159 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:48.891616 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:48.891622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:48.891625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:48.891957 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:48.892110 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:48.892815 (+ 705us) server_negotiation.cc:299] Negotiation successful
0810 01:49:48.893024 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 01:49:48.894263 32476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:48.877033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.130:45535)
0810 01:49:48.877744 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:48.877851 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:48.879241 (+ 1390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:48.880049 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:48.880061 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:48.880691 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:49:48.881711 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:48.881736 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:48.882733 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:48.882740 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:48.884067 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:48.884082 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:48.884689 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:48.884692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:48.884925 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:48.885768 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:49:48.885791 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:49:48.888103 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:49:48.890477 (+ 2374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:48.890484 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:48.890499 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:48.890871 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:48.891275 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:48.891278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:48.891280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:48.891489 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:48.892123 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:49:48.892130 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:49:48.892485 (+ 355us) client_negotiation.cc:769] Sending connection context
0810 01:49:48.892702 (+ 217us) client_negotiation.cc:240] Negotiation successful
0810 01:49:48.893105 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":143,"threads_started":1}
I20260810 01:49:48.895406 32308 tablet_bootstrap.cc:492] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 01:49:48.895769 32473 heartbeater.cc:347] Connected to a master server at 127.9.77.190:46111
I20260810 01:49:48.895962 32308 tablet_bootstrap.cc:492] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97: Bootstrap complete.
I20260810 01:49:48.896131 32473 heartbeater.cc:464] Registering TS with master...
I20260810 01:49:48.896931 32473 heartbeater.cc:511] Master 127.9.77.190:46111 requested a full tablet report, sending...
I20260810 01:49:48.897320 32308 ts_tablet_manager.cc:1397] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97: Time spent bootstrapping tablet: real 0.037s user 0.024s sys 0.012s
I20260810 01:49:48.899936 31976 ts_manager.cc:194] Registered new tserver with Master: 770f1e22d67044aa9525812b24ddaf97 (127.9.77.130:40817)
I20260810 01:49:48.900334 32308 raft_consensus.cc:348] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } }
I20260810 01:49:48.901010 32308 raft_consensus.cc:729] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770f1e22d67044aa9525812b24ddaf97, State: Initialized, Role: FOLLOWER
I20260810 01:49:48.901532 31976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.130@KRBTEST.COM'} at 127.9.77.130:45535
I20260810 01:49:48.901614 32308 consensus_queue.cc:260] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } }
I20260810 01:49:48.902881 32308 ts_tablet_manager.cc:1428] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 01:49:48.904372 32473 heartbeater.cc:503] Master 127.9.77.190:46111 was elected leader, sending a full tablet report...
Aug 10 01:49:48 dist-test-slave-kvfs krb5kdc[31113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326588, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for kudu/127.9.77.130@KRBTEST.COM
I20260810 01:49:48.914142 32480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:48.901751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49584 (local address 127.9.77.130:40817)
0810 01:49:48.902297 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:48.902309 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:48.902334 (+ 25us) server_negotiation.cc:407] Connection header received
0810 01:49:48.902424 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:48.902430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:48.903251 (+ 821us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:48.903416 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:48.903926 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:48.905188 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:48.906519 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:48.907047 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:48.907171 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:48.909918 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:48.909932 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:48.909955 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:48.909982 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:48.911901 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:48.912483 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:48.912489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:48.912491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:48.912579 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:48.912899 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:48.912903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:48.912905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:48.913205 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:48.913390 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:48.913721 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 01:49:48.913911 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"spinlock_wait_cycles":768,"thread_start_us":147,"threads_started":1}
I20260810 01:49:48.914160 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:48.901166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:40817 (local address 127.0.0.1:49584)
0810 01:49:48.901493 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:48.901508 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:48.901599 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:48.903425 (+ 1826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:48.903436 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:48.903674 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:49:48.903795 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:48.903803 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:48.905331 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:48.905335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:48.906401 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:48.906414 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:48.907187 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:48.907191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:48.907309 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:48.907927 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:49:48.907944 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:49:48.909755 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:49:48.912086 (+ 2331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:48.912089 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:48.912092 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:48.912334 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:48.912686 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:48.912690 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:48.912692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:48.912801 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:48.913344 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:49:48.913347 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:49:48.913474 (+ 127us) client_negotiation.cc:769] Sending connection context
0810 01:49:48.913625 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 01:49:48.913916 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233}
I20260810 01:49:48.916256 32376 tablet_service.cc:1514] Processing DeleteTablet for tablet f00e278d4c2b46fea4c799e472d4ce63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:49:48 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:49584
I20260810 01:49:48.917675 32482 tablet_replica.cc:330] stopping tablet replica
I20260810 01:49:48.918143 32482 raft_consensus.cc:2229] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:49:48.918488 32482 raft_consensus.cc:2258] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:49:48.919862 32482 ts_tablet_manager.cc:1905] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:49:48.925693 32482 ts_tablet_manager.cc:1918] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:49:48.925832 32482 log.cc:1196] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/wal/wals/f00e278d4c2b46fea4c799e472d4ce63
I20260810 01:49:48.926152 32482 ts_tablet_manager.cc:1939] T f00e278d4c2b46fea4c799e472d4ce63 P 770f1e22d67044aa9525812b24ddaf97: Deleting consensus metadata
I20260810 01:49:48.928766 31954 catalog_manager.cc:4841] TS 770f1e22d67044aa9525812b24ddaf97 (127.9.77.130:40817): tablet f00e278d4c2b46fea4c799e472d4ce63 (table test-table [id=1f56139e917544c0b845d01e13fc83ab]) successfully deleted
W20260810 01:49:49.031894 32479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:49:49.032292 32479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:49:49.032339 32479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:49:49.036407 32479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:49:49.036468 32479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:49:49.036525 32479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:49:49.041973 32479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:33877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:35807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=33231
--tserver_master_addrs=127.9.77.190:46111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:49:49.043318 32479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:49:49.044678 32479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:49:49.056407 32494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:49.056397 32496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:49.056393 32492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:49.056793 32479 server_base.cc:1034] running on GCE node
I20260810 01:49:49.057315 32479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:49:49.058106 32479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:49:49.059314 32479 hybrid_clock.cc:648] HybridClock initialized: now 1786326589059300 us; error 33 us; skew 500 ppm
Aug 10 01:49:49 dist-test-slave-kvfs krb5kdc[31113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326589, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:49:49.062990 32479 init.cc:377] Logged in from keytab as kudu/127.9.77.131@KRBTEST.COM (short username kudu)
I20260810 01:49:49.064234 32479 webserver.cc:460] Webserver started at http://127.9.77.131:33231/ using document root <none> and password file <none>
I20260810 01:49:49.064990 32479 fs_manager.cc:359] Metadata directory not provided
I20260810 01:49:49.065136 32479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:49:49.070432 32479 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260810 01:49:49.073228 32504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:49.074782 32479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260810 01:49:49.074900 32479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e"
format_stamp: "Formatted at 2026-08-10 01:49:47 on dist-test-slave-kvfs"
server_key: "73da1bacad03c3c7f03817b5327b3992"
server_key_iv: "73078766f5a693c62765dde359ad5262"
server_key_version: "encryptionkey@0"
I20260810 01:49:49.075537 32479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:49:49.094626 32479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:49:49.098718 32479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:49:49.098920 32479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:49:49.099611 32479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:49:49.100653 32479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:49:49.100737 32479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:49.100827 32479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:49:49.100865 32479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:49.113551 32479 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:35807
I20260810 01:49:49.113660 32680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:35807 every 8 connection(s)
I20260810 01:49:49.115010 32479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:49:49.128078 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 32479
Aug 10 01:49:49 dist-test-slave-kvfs krb5kdc[31113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326589, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:49:49.139202 32268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:49.118043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:49317 (local address 127.9.77.190:46111)
0810 01:49:49.118241 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:49.118247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:49.120262 (+ 2015us) server_negotiation.cc:407] Connection header received
0810 01:49:49.121772 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:49.121799 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:49.121872 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:49.121966 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:49.124607 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:49.125619 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:49.127920 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:49.128362 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:49.128469 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:49.132709 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:49.132715 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:49.132719 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:49.132762 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:49.135080 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:49.135988 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:49.135993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:49.135995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:49.136143 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:49.136741 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:49.136745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:49.136747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:49.137067 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:49.137168 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:49.138310 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0810 01:49:49.138915 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 01:49:49.142202 32687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:49.118630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.131:49317)
0810 01:49:49.119932 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:49.120098 (+ 166us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:49.121393 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:49.122428 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:49.122439 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:49.123236 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:49:49.124358 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:49.124389 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:49.125797 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:49.125802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:49.127703 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:49.127719 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:49.128754 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:49.128763 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:49.129001 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:49.129972 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:49:49.130000 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:49:49.132471 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:49:49.135238 (+ 2767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:49.135251 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:49.135292 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:49.135767 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:49.136299 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:49.136323 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:49.136325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:49.136585 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:49.137273 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:49:49.137283 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:49:49.137808 (+ 525us) client_negotiation.cc:769] Sending connection context
0810 01:49:49.138185 (+ 377us) client_negotiation.cc:240] Negotiation successful
0810 01:49:49.140631 (+ 2446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1016,"thread_start_us":841,"threads_started":1}
I20260810 01:49:49.143077 32689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:49.129109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59646 (local address 127.9.77.190:46111)
0810 01:49:49.129972 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:49.129978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:49.129993 (+ 15us) server_negotiation.cc:407] Connection header received
0810 01:49:49.130038 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:49.130041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:49.130105 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:49.130214 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:49.132438 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:49.133233 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:49.134727 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:49.135189 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:49.135292 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:49.137448 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:49.137452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:49.137455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:49.137487 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:49.139880 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:49.140500 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:49.140503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:49.140505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:49.140612 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:49.141035 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:49.141039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:49.141041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:49.141291 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:49.141557 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:49.142775 (+ 1218us) server_negotiation.cc:299] Negotiation successful
0810 01:49:49.142925 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":132,"threads_started":1}
I20260810 01:49:49.144874 32683 heartbeater.cc:347] Connected to a master server at 127.9.77.190:46111
I20260810 01:49:49.145298 32683 heartbeater.cc:464] Registering TS with master...
I20260810 01:49:49.146198 32683 heartbeater.cc:511] Master 127.9.77.190:46111 requested a full tablet report, sending...
I20260810 01:49:49.147738 31976 ts_manager.cc:194] Registered new tserver with Master: 4a02a22a13354c07bb0cde6f7c5b4d4e (127.9.77.131:35807)
I20260810 01:49:49.149091 31976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.131@KRBTEST.COM'} at 127.9.77.131:49317
I20260810 01:49:49.159543 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 01:49:49.167092 32689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:49.160564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59654 (local address 127.9.77.190:46111)
0810 01:49:49.160724 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:49.160729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:49.161051 (+ 322us) server_negotiation.cc:407] Connection header received
0810 01:49:49.161371 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:49.161380 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:49.161622 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:49.161792 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:49:49.162178 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:49.163035 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:49.164749 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:49.165167 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:49.165291 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:49.165809 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:49:49.166554 (+ 745us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:49:49.166688 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:49.166808 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 01:49:49.166893 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 01:49:49.167902 31976 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:59654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 01:49:49.172514 31976 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:49:49.182425 32376 tablet_service.cc:1467] Processing CreateTablet for tablet 097ea812b3764b95bc781c17e6f2f808 (DEFAULT_TABLE table=test-table [id=3d6bd636539647efa1430bbfd48e428c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:49:49.183246 32376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 097ea812b3764b95bc781c17e6f2f808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:49:49.185075 32163 tablet_service.cc:1467] Processing CreateTablet for tablet 097ea812b3764b95bc781c17e6f2f808 (DEFAULT_TABLE table=test-table [id=3d6bd636539647efa1430bbfd48e428c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:49:49.185801 32163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 097ea812b3764b95bc781c17e6f2f808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:49:49.188236 32308 tablet_bootstrap.cc:492] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97: Bootstrap starting.
I20260810 01:49:49.189244 32697 tablet_bootstrap.cc:492] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a: Bootstrap starting.
I20260810 01:49:49.189302 32308 tablet_bootstrap.cc:654] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97: Neither blocks nor log segments found. Creating new log.
I20260810 01:49:49.190362 32697 tablet_bootstrap.cc:654] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a: Neither blocks nor log segments found. Creating new log.
I20260810 01:49:49.190970 32308 tablet_bootstrap.cc:492] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97: No bootstrap required, opened a new log
I20260810 01:49:49.191077 32308 ts_tablet_manager.cc:1397] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260810 01:49:49.191607 32308 raft_consensus.cc:348] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } }
I20260810 01:49:49.191708 32697 tablet_bootstrap.cc:492] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a: No bootstrap required, opened a new log
I20260810 01:49:49.191749 32308 raft_consensus.cc:374] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:49:49.191789 32308 raft_consensus.cc:729] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770f1e22d67044aa9525812b24ddaf97, State: Initialized, Role: FOLLOWER
I20260810 01:49:49.191841 32697 ts_tablet_manager.cc:1397] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 01:49:49.192233 32308 consensus_queue.cc:260] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } }
I20260810 01:49:49.192404 32697 raft_consensus.cc:348] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } }
I20260810 01:49:49.192529 32697 raft_consensus.cc:374] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:49:49.192569 32697 raft_consensus.cc:729] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4562ab42d4461e84c13b245eec5b5a, State: Initialized, Role: FOLLOWER
I20260810 01:49:49.192729 32697 consensus_queue.cc:260] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } }
I20260810 01:49:49.192854 32308 ts_tablet_manager.cc:1428] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260810 01:49:49.193097 32697 ts_tablet_manager.cc:1428] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 01:49:49.194223 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:49.181169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:35807 (local address 127.0.0.1:49090)
0810 01:49:49.181539 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:49.181557 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:49.181652 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:49.182598 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:49.182602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:49.182624 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:49:49.182751 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:49.182757 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:49.190547 (+ 7790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:49.190551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:49.191965 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:49.191984 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:49.193692 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:49.193695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:49.193799 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:49.193862 (+ 63us) client_negotiation.cc:769] Sending connection context
0810 01:49:49.193944 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 01:49:49.194056 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284}
I20260810 01:49:49.194936 32696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:49.181433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49090 (local address 127.9.77.131:35807)
0810 01:49:49.181899 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:49.181918 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:49.181952 (+ 34us) server_negotiation.cc:407] Connection header received
0810 01:49:49.182068 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:49.182076 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:49.182397 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:49.182642 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:49:49.182841 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:49.184303 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:49.192194 (+ 7891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:49.193373 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:49.193535 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:49.193772 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:49.194615 (+ 843us) server_negotiation.cc:299] Negotiation successful
0810 01:49:49.194763 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":186,"threads_started":1}
I20260810 01:49:49.199034 32580 tablet_service.cc:1467] Processing CreateTablet for tablet 097ea812b3764b95bc781c17e6f2f808 (DEFAULT_TABLE table=test-table [id=3d6bd636539647efa1430bbfd48e428c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:49:49.200366 32580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 097ea812b3764b95bc781c17e6f2f808. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:49:49.209265 32703 tablet_bootstrap.cc:492] T 097ea812b3764b95bc781c17e6f2f808 P 4a02a22a13354c07bb0cde6f7c5b4d4e: Bootstrap starting.
I20260810 01:49:49.212544 32703 tablet_bootstrap.cc:654] T 097ea812b3764b95bc781c17e6f2f808 P 4a02a22a13354c07bb0cde6f7c5b4d4e: Neither blocks nor log segments found. Creating new log.
I20260810 01:49:49.213478 32703 log.cc:824] T 097ea812b3764b95bc781c17e6f2f808 P 4a02a22a13354c07bb0cde6f7c5b4d4e: Log is configured to *not* fsync() on all Append() calls
I20260810 01:49:49.215740 32703 tablet_bootstrap.cc:492] T 097ea812b3764b95bc781c17e6f2f808 P 4a02a22a13354c07bb0cde6f7c5b4d4e: No bootstrap required, opened a new log
I20260810 01:49:49.216058 32703 ts_tablet_manager.cc:1397] T 097ea812b3764b95bc781c17e6f2f808 P 4a02a22a13354c07bb0cde6f7c5b4d4e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 01:49:49.219512 32703 raft_consensus.cc:348] T 097ea812b3764b95bc781c17e6f2f808 P 4a02a22a13354c07bb0cde6f7c5b4d4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } }
I20260810 01:49:49.219836 32703 raft_consensus.cc:374] T 097ea812b3764b95bc781c17e6f2f808 P 4a02a22a13354c07bb0cde6f7c5b4d4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:49:49.219902 32703 raft_consensus.cc:729] T 097ea812b3764b95bc781c17e6f2f808 P 4a02a22a13354c07bb0cde6f7c5b4d4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a02a22a13354c07bb0cde6f7c5b4d4e, State: Initialized, Role: FOLLOWER
I20260810 01:49:49.220454 32703 consensus_queue.cc:260] T 097ea812b3764b95bc781c17e6f2f808 P 4a02a22a13354c07bb0cde6f7c5b4d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } }
I20260810 01:49:49.221261 32683 heartbeater.cc:503] Master 127.9.77.190:46111 was elected leader, sending a full tablet report...
I20260810 01:49:49.221633 32703 ts_tablet_manager.cc:1428] T 097ea812b3764b95bc781c17e6f2f808 P 4a02a22a13354c07bb0cde6f7c5b4d4e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 01:49:49.322263 32481 raft_consensus.cc:482] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:49:49.322706 32481 raft_consensus.cc:504] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } }
I20260810 01:49:49.324368 32481 leader_election.cc:288] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a02a22a13354c07bb0cde6f7c5b4d4e (127.9.77.131:35807), 1b4562ab42d4461e84c13b245eec5b5a (127.9.77.129:45863)
I20260810 01:49:49.329236 32710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:49.324847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:55779 (local address 127.9.77.129:45863)
0810 01:49:49.325215 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:49.325220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:49.325300 (+ 80us) server_negotiation.cc:407] Connection header received
0810 01:49:49.325451 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:49.325454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:49.325534 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:49.325660 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:49:49.326025 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:49.326657 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:49.327786 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:49.328496 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:49.328644 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:49.328841 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:49.328994 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 01:49:49.329067 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":113,"threads_started":1}
I20260810 01:49:49.329232 32696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:49.324839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:41069 (local address 127.9.77.131:35807)
0810 01:49:49.324982 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:49.324987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:49.325005 (+ 18us) server_negotiation.cc:407] Connection header received
0810 01:49:49.325241 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:49.325247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:49.325325 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:49.325409 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:49:49.325784 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:49.326456 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:49.327750 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:49.328384 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:49.328539 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:49.328613 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:49.328988 (+ 375us) server_negotiation.cc:299] Negotiation successful
0810 01:49:49.329081 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 01:49:49.329231 32476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:49.324669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:35807 (local address 127.9.77.130:41069)
0810 01:49:49.324869 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:49.324888 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:49.325022 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:49.325438 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:49.325442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:49.325464 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:49:49.325650 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:49.325658 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:49.326580 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:49.326583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:49.327600 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:49.327610 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:49.328524 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:49.328528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:49.328641 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:49.328856 (+ 215us) client_negotiation.cc:769] Sending connection context
0810 01:49:49.328966 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 01:49:49.329059 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":136}
I20260810 01:49:49.329258 32711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:49.324668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:45863 (local address 127.9.77.130:55779)
0810 01:49:49.325185 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:49.325199 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:49.325320 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:49.325676 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:49.325678 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:49.325693 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:49:49.325881 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:49.325888 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:49.326780 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:49.326783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:49.327619 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:49.327628 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:49.328639 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:49.328642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:49.328792 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:49.328879 (+ 87us) client_negotiation.cc:769] Sending connection context
0810 01:49:49.328956 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 01:49:49.329052 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":215,"thread_start_us":125,"threads_started":1}
I20260810 01:49:49.330089 32197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "097ea812b3764b95bc781c17e6f2f808" candidate_uuid: "770f1e22d67044aa9525812b24ddaf97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4562ab42d4461e84c13b245eec5b5a" is_pre_election: true
I20260810 01:49:49.330174 32613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "097ea812b3764b95bc781c17e6f2f808" candidate_uuid: "770f1e22d67044aa9525812b24ddaf97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" is_pre_election: true
I20260810 01:49:49.330529 32197 raft_consensus.cc:2454] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 770f1e22d67044aa9525812b24ddaf97 in term 0.
I20260810 01:49:49.330528 32613 raft_consensus.cc:2454] T 097ea812b3764b95bc781c17e6f2f808 P 4a02a22a13354c07bb0cde6f7c5b4d4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 770f1e22d67044aa9525812b24ddaf97 in term 0.
I20260810 01:49:49.331095 32302 leader_election.cc:302] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1b4562ab42d4461e84c13b245eec5b5a, 4a02a22a13354c07bb0cde6f7c5b4d4e, 770f1e22d67044aa9525812b24ddaf97; no voters:
I20260810 01:49:49.331431 32481 raft_consensus.cc:2781] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:49:49.331527 32481 raft_consensus.cc:482] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:49:49.331583 32481 raft_consensus.cc:3037] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:49:49.333000 32481 raft_consensus.cc:504] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } }
I20260810 01:49:49.333420 32481 leader_election.cc:288] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97 [CANDIDATE]: Term 1 election: Requested vote from peers 4a02a22a13354c07bb0cde6f7c5b4d4e (127.9.77.131:35807), 1b4562ab42d4461e84c13b245eec5b5a (127.9.77.129:45863)
I20260810 01:49:49.333806 32613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "097ea812b3764b95bc781c17e6f2f808" candidate_uuid: "770f1e22d67044aa9525812b24ddaf97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e"
I20260810 01:49:49.333811 32197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "097ea812b3764b95bc781c17e6f2f808" candidate_uuid: "770f1e22d67044aa9525812b24ddaf97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4562ab42d4461e84c13b245eec5b5a"
I20260810 01:49:49.333961 32197 raft_consensus.cc:3037] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:49:49.333964 32613 raft_consensus.cc:3037] T 097ea812b3764b95bc781c17e6f2f808 P 4a02a22a13354c07bb0cde6f7c5b4d4e [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:49:49.335456 32197 raft_consensus.cc:2454] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 770f1e22d67044aa9525812b24ddaf97 in term 1.
I20260810 01:49:49.335456 32613 raft_consensus.cc:2454] T 097ea812b3764b95bc781c17e6f2f808 P 4a02a22a13354c07bb0cde6f7c5b4d4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 770f1e22d67044aa9525812b24ddaf97 in term 1.
I20260810 01:49:49.335806 32301 leader_election.cc:302] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a02a22a13354c07bb0cde6f7c5b4d4e, 770f1e22d67044aa9525812b24ddaf97; no voters:
I20260810 01:49:49.336050 32481 raft_consensus.cc:2781] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:49:49.336337 32481 raft_consensus.cc:686] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97 [term 1 LEADER]: Becoming Leader. State: Replica: 770f1e22d67044aa9525812b24ddaf97, State: Running, Role: LEADER
I20260810 01:49:49.336722 32481 consensus_queue.cc:237] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } }
I20260810 01:49:49.340368 31972 catalog_manager.cc:5495] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 770f1e22d67044aa9525812b24ddaf97 (127.9.77.130). New cstate: current_term: 1 leader_uuid: "770f1e22d67044aa9525812b24ddaf97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } health_report { overall_health: UNKNOWN } } }
I20260810 01:49:49.356225 32480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:49.351732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49600 (local address 127.9.77.130:40817)
0810 01:49:49.351900 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:49.351905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:49.351920 (+ 15us) server_negotiation.cc:407] Connection header received
0810 01:49:49.352141 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:49.352145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:49.352227 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:49.352361 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:49:49.352611 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:49.353520 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:49.354392 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:49.354776 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:49.354892 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:49.355150 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:49:49.355800 (+ 650us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:49:49.355920 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:49.355962 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 01:49:49.356061 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 01:49:49.361135 32613 raft_consensus.cc:1261] T 097ea812b3764b95bc781c17e6f2f808 P 4a02a22a13354c07bb0cde6f7c5b4d4e [term 1 FOLLOWER]: Refusing update from remote peer 770f1e22d67044aa9525812b24ddaf97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:49:49.361129 32197 raft_consensus.cc:1261] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [term 1 FOLLOWER]: Refusing update from remote peer 770f1e22d67044aa9525812b24ddaf97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:49:49.361996 32712 consensus_queue.cc:1035] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:49:49.362202 32481 consensus_queue.cc:1035] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 01:49:49.366168 32267 tablet.cc:2369] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 01:49:49.366655 32686 tablet.cc:2369] T 097ea812b3764b95bc781c17e6f2f808 P 4a02a22a13354c07bb0cde6f7c5b4d4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 01:49:49.369371 32715 mvcc.cc:205] Tried to move back new op lower bound from 7316793710014947328 to 7316793709926182912. Current Snapshot: MvccSnapshot[applied={T|T < 7316793710014947328}]
I20260810 01:49:49.379505 31973 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:59654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:49:49.379766 31973 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:59654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:49:49.382652 31973 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55: Sending DeleteTablet for 3 replicas of tablet 097ea812b3764b95bc781c17e6f2f808
I20260810 01:49:49.383540 32376 tablet_service.cc:1514] Processing DeleteTablet for tablet 097ea812b3764b95bc781c17e6f2f808 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:49:49 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:49584
I20260810 01:49:49.383484 32580 tablet_service.cc:1514] Processing DeleteTablet for tablet 097ea812b3764b95bc781c17e6f2f808 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:49:49 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:49090
I20260810 01:49:49.383740 32482 tablet_replica.cc:330] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97: stopping tablet replica
I20260810 01:49:49.383908 32482 raft_consensus.cc:2229] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97 [term 1 LEADER]: Raft consensus shutting down.
I20260810 01:49:49.384038 32163 tablet_service.cc:1514] Processing DeleteTablet for tablet 097ea812b3764b95bc781c17e6f2f808 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:49:49 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:54012
I20260810 01:49:49.384256 32727 tablet_replica.cc:330] T 097ea812b3764b95bc781c17e6f2f808 P 4a02a22a13354c07bb0cde6f7c5b4d4e: stopping tablet replica
I20260810 01:49:49.384302 32482 raft_consensus.cc:2258] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:49:49.384658 32727 raft_consensus.cc:2229] T 097ea812b3764b95bc781c17e6f2f808 P 4a02a22a13354c07bb0cde6f7c5b4d4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:49:49.384795 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 32036
I20260810 01:49:49.385040 32727 raft_consensus.cc:2258] T 097ea812b3764b95bc781c17e6f2f808 P 4a02a22a13354c07bb0cde6f7c5b4d4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:49:49.394424 32482 ts_tablet_manager.cc:1905] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:49:49.399035 32482 ts_tablet_manager.cc:1918] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:49:49.399164 32482 log.cc:1196] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/wal/wals/097ea812b3764b95bc781c17e6f2f808
I20260810 01:49:49.399538 32482 ts_tablet_manager.cc:1939] T 097ea812b3764b95bc781c17e6f2f808 P 770f1e22d67044aa9525812b24ddaf97: Deleting consensus metadata
I20260810 01:49:49.403525 31954 catalog_manager.cc:4841] TS 770f1e22d67044aa9525812b24ddaf97 (127.9.77.130:40817): tablet 097ea812b3764b95bc781c17e6f2f808 (table test-table [id=3d6bd636539647efa1430bbfd48e428c]) successfully deleted
I20260810 01:49:49.417730 32727 ts_tablet_manager.cc:1905] T 097ea812b3764b95bc781c17e6f2f808 P 4a02a22a13354c07bb0cde6f7c5b4d4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:49:49.420787 32727 ts_tablet_manager.cc:1918] T 097ea812b3764b95bc781c17e6f2f808 P 4a02a22a13354c07bb0cde6f7c5b4d4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:49:49.420971 32727 log.cc:1196] T 097ea812b3764b95bc781c17e6f2f808 P 4a02a22a13354c07bb0cde6f7c5b4d4e: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/wal/wals/097ea812b3764b95bc781c17e6f2f808
I20260810 01:49:49.421358 32727 ts_tablet_manager.cc:1939] T 097ea812b3764b95bc781c17e6f2f808 P 4a02a22a13354c07bb0cde6f7c5b4d4e: Deleting consensus metadata
I20260810 01:49:49.423127 31952 catalog_manager.cc:4841] TS 4a02a22a13354c07bb0cde6f7c5b4d4e (127.9.77.131:35807): tablet 097ea812b3764b95bc781c17e6f2f808 (table test-table [id=3d6bd636539647efa1430bbfd48e428c]) successfully deleted
I20260810 01:49:49.425896 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 32270
W20260810 01:49:49.425863 31953 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.77.129:45863 (error 108)
I20260810 01:49:49.432775 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:49.432368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:45863 (local address 127.0.0.1:54018)
0810 01:49:49.432550 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:49.432646 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.129:45863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 01:49:49.433178 31953 catalog_manager.cc:4584] TS 1b4562ab42d4461e84c13b245eec5b5a (127.9.77.129:45863): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 097ea812b3764b95bc781c17e6f2f808: Network error: Client connection negotiation failed: client connection to 127.9.77.129:45863: connect: Connection refused (error 111)
I20260810 01:49:49.439939 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 32479
I20260810 01:49:49.456828 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 31932
I20260810 01:49:49.473183 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:46111
--webserver_interface=127.9.77.190
--webserver_port=39865
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:33877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:46111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:49:49.609287 32731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:49:49.609710 32731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:49:49.609793 32731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:49:49.613952 32731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:49:49.614060 32731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:49:49.614096 32731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:49:49.614166 32731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:49:49.614231 32731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:49:49.619778 32731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:33877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:46111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:46111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=39865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:49:49.621215 32731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:49:49.622541 32731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:49:49.633024 32745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:49.633071 32748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:49.633175 32731 server_base.cc:1034] running on GCE node
W20260810 01:49:49.633072 32744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:49.634094 32731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:49:49.635452 32731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:49:49.636651 32731 hybrid_clock.cc:648] HybridClock initialized: now 1786326589636638 us; error 27 us; skew 500 ppm
Aug 10 01:49:49 dist-test-slave-kvfs krb5kdc[31113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326589, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:49:49.639943 32731 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:49:49.641148 32731 webserver.cc:460] Webserver started at http://127.9.77.190:39865/ using document root <none> and password file <none>
I20260810 01:49:49.641855 32731 fs_manager.cc:359] Metadata directory not provided
I20260810 01:49:49.641956 32731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:49:49.647033 32731 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 01:49:49.649839 32762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:49.651544 32731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 01:49:49.651731 32731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "86545495916040a7803b97b6b4d81c55"
format_stamp: "Formatted at 2026-08-10 01:49:46 on dist-test-slave-kvfs"
server_key: "5afa6600abc93e75b96796e193aef87b"
server_key_iv: "a2292fbc78e7e8af0d4be37ca7052b57"
server_key_version: "encryptionkey@0"
I20260810 01:49:49.652554 32731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:49:49.676270 32731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:49:49.679965 32731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:49:49.680251 32731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:49:49.689494 32731 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:46111
I20260810 01:49:49.689857 379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:46111 every 8 connection(s)
I20260810 01:49:49.691056 32731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:49:49.692507 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 32731
I20260810 01:49:49.693101 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:45863
--local_ip_for_outbound_sockets=127.9.77.129
--tserver_master_addrs=127.9.77.190:46111
--webserver_port=39747
--webserver_interface=127.9.77.129
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:33877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 01:49:49.698055 380 sys_catalog.cc:263] Verifying existing consensus state
I20260810 01:49:49.700356 380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55: Bootstrap starting.
I20260810 01:49:49.721742 380 log.cc:824] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55: Log is configured to *not* fsync() on all Append() calls
I20260810 01:49:49.735131 380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 01:49:49.735608 380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55: Bootstrap complete.
I20260810 01:49:49.740451 380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86545495916040a7803b97b6b4d81c55" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46111 } }
I20260810 01:49:49.740913 380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86545495916040a7803b97b6b4d81c55, State: Initialized, Role: FOLLOWER
I20260810 01:49:49.741560 380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86545495916040a7803b97b6b4d81c55" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46111 } }
I20260810 01:49:49.741721 380 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:49:49.741794 380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:49:49.741933 380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [term 2 FOLLOWER]: Advancing to term 3
I20260810 01:49:49.744287 380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86545495916040a7803b97b6b4d81c55" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46111 } }
I20260810 01:49:49.744684 380 leader_election.cc:302] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86545495916040a7803b97b6b4d81c55; no voters:
I20260810 01:49:49.745087 380 leader_election.cc:288] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 01:49:49.745191 385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 01:49:49.745425 385 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [term 3 LEADER]: Becoming Leader. State: Replica: 86545495916040a7803b97b6b4d81c55, State: Running, Role: LEADER
I20260810 01:49:49.745759 385 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86545495916040a7803b97b6b4d81c55" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46111 } }
I20260810 01:49:49.746589 380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:49:49.747355 387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "86545495916040a7803b97b6b4d81c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86545495916040a7803b97b6b4d81c55" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46111 } } }
I20260810 01:49:49.747529 387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [sys.catalog]: This master's current role is: LEADER
I20260810 01:49:49.747865 391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86545495916040a7803b97b6b4d81c55. Latest consensus state: current_term: 3 leader_uuid: "86545495916040a7803b97b6b4d81c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86545495916040a7803b97b6b4d81c55" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46111 } } }
I20260810 01:49:49.748051 391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86545495916040a7803b97b6b4d81c55 [sys.catalog]: This master's current role is: LEADER
I20260810 01:49:49.748773 396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:49:49.754736 396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:49:49.755445 396 catalog_manager.cc:1259] Loaded cluster ID: a5b5a3d2f58e40ffaaac013f261a3ac8
I20260810 01:49:49.755510 396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 01:49:49.756822 396 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2cf94319ad google::LogMessage::Fail() at ??:0
@ 0x7f2cf943488a google::LogMessage::SendToLog() at ??:0
@ 0x7f2cf94314de google::LogMessage::Flush() at ??:0
@ 0x7f2cf9432a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2d03d74fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f2d03d75d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2d03d72579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f2d03db6640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f2cfb6833d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2cfb683c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f2cfb6853f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f2cfb674512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2cfccb86db start_thread at ??:0
@ 0x7f2cf7f2f61f clone at ??:0
W20260810 01:49:49.848799 382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:49:49.849180 382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:49:49.849287 382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:49:49.853832 382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:49:49.853935 382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:49:49.854065 382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:49:49.860304 382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:33877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:45863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=39747
--tserver_master_addrs=127.9.77.190:46111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:49:49.861826 382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:49:49.863292 382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:49:49.875573 438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:49.875552 439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:49.875869 446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:49.875854 382 server_base.cc:1034] running on GCE node
I20260810 01:49:49.876458 382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:49:49.877305 382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:49:49.878501 382 hybrid_clock.cc:648] HybridClock initialized: now 1786326589878474 us; error 54 us; skew 500 ppm
Aug 10 01:49:49 dist-test-slave-kvfs krb5kdc[31113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326589, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:49:49.881920 382 init.cc:377] Logged in from keytab as kudu/127.9.77.129@KRBTEST.COM (short username kudu)
I20260810 01:49:49.883147 382 webserver.cc:460] Webserver started at http://127.9.77.129:39747/ using document root <none> and password file <none>
I20260810 01:49:49.883849 382 fs_manager.cc:359] Metadata directory not provided
I20260810 01:49:49.883961 382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:49:49.888845 382 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:49:49.891508 457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:49.893211 382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:49:49.893365 382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "1b4562ab42d4461e84c13b245eec5b5a"
format_stamp: "Formatted at 2026-08-10 01:49:47 on dist-test-slave-kvfs"
server_key: "45820d42824fbe0a353bcb7c0d7c4221"
server_key_iv: "79be2319cfc89e2198de4ea0df445122"
server_key_version: "encryptionkey@0"
I20260810 01:49:49.894060 382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:49:49.925225 382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:49:49.929744 382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:49:49.930004 382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:49:49.930860 382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:49:49.932324 473 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 01:49:49.937641 382 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 01:49:49.937738 382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260810 01:49:49.937832 382 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 01:49:49.940013 382 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 01:49:49.940099 382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.002s
I20260810 01:49:49.940464 473 tablet_bootstrap.cc:492] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a: Bootstrap starting.
I20260810 01:49:49.953061 382 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:45863
I20260810 01:49:49.953474 620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:45863 every 8 connection(s)
I20260810 01:49:49.954829 382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:49:49.964339 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 382
I20260810 01:49:49.964951 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:40817
--local_ip_for_outbound_sockets=127.9.77.130
--tserver_master_addrs=127.9.77.190:46111
--webserver_port=38535
--webserver_interface=127.9.77.130
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:33877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 01:49:49.965752 473 log.cc:824] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a: Log is configured to *not* fsync() on all Append() calls
I20260810 01:49:49.976894 473 tablet_bootstrap.cc:492] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 01:49:49.977478 473 tablet_bootstrap.cc:492] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a: Bootstrap complete.
I20260810 01:49:49.979187 473 ts_tablet_manager.cc:1397] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a: Time spent bootstrapping tablet: real 0.039s user 0.026s sys 0.009s
I20260810 01:49:49.984086 473 raft_consensus.cc:348] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } }
I20260810 01:49:49.984860 473 raft_consensus.cc:729] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4562ab42d4461e84c13b245eec5b5a, State: Initialized, Role: FOLLOWER
I20260810 01:49:49.985582 473 consensus_queue.cc:260] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } }
I20260810 01:49:49.996179 473 ts_tablet_manager.cc:1428] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a: Time spent starting tablet: real 0.017s user 0.011s sys 0.000s
W20260810 01:49:50.003934 630 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:49:49.958103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.129:59339)
0810 01:49:49.958871 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:49.958945 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:49.960226 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:50.002897 (+ 42671us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":222,"threads_started":1}
W20260810 01:49:50.004590 625 heartbeater.cc:644] Failed to heartbeat to 127.9.77.190:46111 (0 consecutive failures): Network error: Failed to ping master at 127.9.77.190:46111: Client connection negotiation failed: client connection to 127.9.77.190:46111: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 01:49:50.005483 630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:50.005190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.129:42973)
0810 01:49:50.005335 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:50.005421 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 01:49:50.006177 630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:50.005975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.129:52899)
0810 01:49:50.006076 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:50.006130 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 01:49:50.006353 625 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 01:49:50.112461 632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:49:50.112820 632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:49:50.112905 632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:49:50.117054 632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:49:50.117151 632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:49:50.117242 632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:49:50.122745 632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:33877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:40817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=38535
--tserver_master_addrs=127.9.77.190:46111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:49:50.124116 632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:49:50.125483 632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:49:50.136878 641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:50.136976 642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:50.136976 646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:50.137111 632 server_base.cc:1034] running on GCE node
I20260810 01:49:50.137717 632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:49:50.138520 632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:49:50.139720 632 hybrid_clock.cc:648] HybridClock initialized: now 1786326590139685 us; error 49 us; skew 500 ppm
Aug 10 01:49:50 dist-test-slave-kvfs krb5kdc[31113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326590, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:49:50.145992 632 init.cc:377] Logged in from keytab as kudu/127.9.77.130@KRBTEST.COM (short username kudu)
I20260810 01:49:50.147006 632 webserver.cc:460] Webserver started at http://127.9.77.130:38535/ using document root <none> and password file <none>
I20260810 01:49:50.147662 632 fs_manager.cc:359] Metadata directory not provided
I20260810 01:49:50.147727 632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:49:50.152359 632 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 01:49:50.154765 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:50.156356 632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:49:50.156487 632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "770f1e22d67044aa9525812b24ddaf97"
format_stamp: "Formatted at 2026-08-10 01:49:47 on dist-test-slave-kvfs"
server_key: "890bde52285b2c23447f6e274877535c"
server_key_iv: "296486718a1f69d42350193f42cd8e31"
server_key_version: "encryptionkey@0"
I20260810 01:49:50.157157 632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:49:50.183837 632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:49:50.187568 632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:49:50.187822 632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:49:50.188643 632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:49:50.189707 632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:49:50.189757 632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:50.189803 632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:49:50.189826 632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:50.201189 632 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:40817
I20260810 01:49:50.201211 814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:40817 every 8 connection(s)
I20260810 01:49:50.202464 632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:49:50.207350 821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:50.205091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.130:49089)
0810 01:49:50.205690 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:50.205824 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":134,"threads_started":1}
W20260810 01:49:50.208005 815 heartbeater.cc:644] Failed to heartbeat to 127.9.77.190:46111 (0 consecutive failures): Network error: Failed to ping master at 127.9.77.190:46111: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
I20260810 01:49:50.208752 821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:50.208517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.130:55899)
0810 01:49:50.208634 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:50.208688 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 01:49:50.209425 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 632
I20260810 01:49:50.209467 821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:50.209257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.130:51059)
0810 01:49:50.209362 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:50.209405 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 01:49:50.209651 815 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 01:49:50.210019 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:35807
--local_ip_for_outbound_sockets=127.9.77.131
--tserver_master_addrs=127.9.77.190:46111
--webserver_port=33231
--webserver_interface=127.9.77.131
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:33877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:49:50.347016 822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:49:50.347534 822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:49:50.347613 822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:49:50.351729 822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:49:50.351840 822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:49:50.351953 822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:49:50.357823 822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:33877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:35807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=33231
--tserver_master_addrs=127.9.77.190:46111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:49:50.359308 822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:49:50.360688 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:49:50.372521 828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:50.372658 822 server_base.cc:1034] running on GCE node
W20260810 01:49:50.372521 834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:49:50.372521 830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:49:50.373438 822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:49:50.374241 822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:49:50.375427 822 hybrid_clock.cc:648] HybridClock initialized: now 1786326590375382 us; error 56 us; skew 500 ppm
Aug 10 01:49:50 dist-test-slave-kvfs krb5kdc[31113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326590, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:49:50.378729 822 init.cc:377] Logged in from keytab as kudu/127.9.77.131@KRBTEST.COM (short username kudu)
I20260810 01:49:50.379877 822 webserver.cc:460] Webserver started at http://127.9.77.131:33231/ using document root <none> and password file <none>
I20260810 01:49:50.380746 822 fs_manager.cc:359] Metadata directory not provided
I20260810 01:49:50.380859 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:49:50.385563 822 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:49:50.388083 841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:50.389683 822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:49:50.389828 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e"
format_stamp: "Formatted at 2026-08-10 01:49:47 on dist-test-slave-kvfs"
server_key: "73da1bacad03c3c7f03817b5327b3992"
server_key_iv: "73078766f5a693c62765dde359ad5262"
server_key_version: "encryptionkey@0"
I20260810 01:49:50.390525 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:49:50.409199 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:49:50.412969 822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:49:50.413216 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:49:50.413988 822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:49:50.415019 822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:49:50.415095 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:50.415180 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:49:50.415217 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:49:50.426304 822 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:35807
I20260810 01:49:50.426362 1008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:35807 every 8 connection(s)
I20260810 01:49:50.427675 822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:49:50.430193 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 822
I20260810 01:49:50.433025 1013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:50.430596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.131:32841)
0810 01:49:50.431229 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:50.431386 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":174,"threads_started":1}
W20260810 01:49:50.433530 1010 heartbeater.cc:644] Failed to heartbeat to 127.9.77.190:46111 (0 consecutive failures): Network error: Failed to ping master at 127.9.77.190:46111: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
I20260810 01:49:50.434212 1013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:50.433976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.131:56299)
0810 01:49:50.434094 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:50.434153 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 01:49:50.434813 1013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:50.434598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.131:44729)
0810 01:49:50.434705 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:50.434752 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 01:49:50.435014 1010 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 01:49:51.007752 1015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:51.007130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.129:54903)
0810 01:49:51.007496 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:51.007573 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":102,"threads_started":1}
I20260810 01:49:51.211030 1017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:51.210457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.130:39515)
0810 01:49:51.210807 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:51.210880 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":96,"threads_started":1}
I20260810 01:49:51.436179 1018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:51.435619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.131:50577)
0810 01:49:51.435911 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:51.435972 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":92,"threads_started":1}
I20260810 01:49:51.537398 1019 raft_consensus.cc:482] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:49:51.537740 1019 raft_consensus.cc:504] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } }
I20260810 01:49:51.539327 1019 leader_election.cc:288] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a02a22a13354c07bb0cde6f7c5b4d4e (127.9.77.131:35807), 770f1e22d67044aa9525812b24ddaf97 (127.9.77.130:40817)
Aug 10 01:49:51 dist-test-slave-kvfs krb5kdc[31113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326589, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for kudu/127.9.77.130@KRBTEST.COM
Aug 10 01:49:51 dist-test-slave-kvfs krb5kdc[31113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326589, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for kudu/127.9.77.131@KRBTEST.COM
I20260810 01:49:51.558172 1023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:51.539873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:32803 (local address 127.9.77.130:40817)
0810 01:49:51.540467 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:51.540494 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:51.540542 (+ 48us) server_negotiation.cc:407] Connection header received
0810 01:49:51.541380 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:51.541387 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:51.541839 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:51.542176 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:51.543704 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:51.545415 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:51.547613 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:51.548450 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:51.548647 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:51.552642 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:51.552653 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:51.552675 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:51.552724 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:51.555105 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:51.555785 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:51.555792 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:51.555798 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:51.555950 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:51.556341 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:51.556345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:51.556346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:51.556897 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:51.557052 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:51.557637 (+ 585us) server_negotiation.cc:299] Negotiation successful
0810 01:49:51.557939 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":173,"threads_started":1}
I20260810 01:49:51.558172 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:51.539652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:40817 (local address 127.9.77.129:32803)
0810 01:49:51.540094 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:51.540109 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:51.540175 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:51.542666 (+ 2491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:51.542668 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:51.542950 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:49:51.543502 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:51.543520 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:51.545624 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:51.545628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:51.547471 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:51.547483 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:51.548602 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:51.548605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:51.548911 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:51.549812 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:49:51.549857 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:49:51.552238 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:49:51.555267 (+ 3029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:51.555274 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:51.555291 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:51.555655 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:51.556078 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:51.556083 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:51.556087 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:51.556237 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:51.557032 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:49:51.557040 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:49:51.557279 (+ 239us) client_negotiation.cc:769] Sending connection context
0810 01:49:51.557494 (+ 215us) client_negotiation.cc:240] Negotiation successful
0810 01:49:51.557905 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":363,"spinlock_wait_cycles":51968,"thread_start_us":98,"threads_started":1}
I20260810 01:49:51.558216 1024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:51.539967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:39805 (local address 127.9.77.131:35807)
0810 01:49:51.540467 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0810 01:49:51.540494 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 01:49:51.540542 (+ 48us) server_negotiation.cc:407] Connection header received
0810 01:49:51.541379 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:49:51.541386 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:49:51.541838 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:49:51.542180 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:49:51.543728 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:51.545400 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:51.547613 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:49:51.548463 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:49:51.548647 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:51.552902 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:49:51.552907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:49:51.552919 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:49:51.552954 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:49:51.555167 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:51.556136 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:51.556141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:51.556145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:51.556277 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:49:51.556610 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:49:51.556613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:49:51.556615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:49:51.557047 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:49:51.557191 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 01:49:51.557640 (+ 449us) server_negotiation.cc:299] Negotiation successful
0810 01:49:51.558004 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":154,"threads_started":1}
I20260810 01:49:51.558199 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:51.539573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:35807 (local address 127.9.77.129:39805)
0810 01:49:51.539965 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:51.539984 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 01:49:51.540156 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:49:51.542275 (+ 2119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:49:51.542283 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:49:51.542928 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:49:51.543501 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:51.543519 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:51.545624 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:51.545628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:51.547472 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:49:51.547483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:49:51.548601 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:49:51.548605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:49:51.548907 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:49:51.549812 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:49:51.549843 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:49:51.552676 (+ 2833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:49:51.555359 (+ 2683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:51.555362 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:51.555364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:51.555659 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:51.556395 (+ 736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:49:51.556398 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:49:51.556402 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:49:51.556501 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:49:51.557174 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:49:51.557177 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:49:51.557280 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 01:49:51.557494 (+ 214us) client_negotiation.cc:240] Negotiation successful
0810 01:49:51.558004 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":461,"spinlock_wait_cycles":36608,"thread_start_us":103,"threads_started":1}
W20260810 01:49:51.561879 459 leader_election.cc:341] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4a02a22a13354c07bb0cde6f7c5b4d4e (127.9.77.131:35807): Not found: Tablet not found: 097ea812b3764b95bc781c17e6f2f808
W20260810 01:49:51.562157 466 leader_election.cc:341] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 770f1e22d67044aa9525812b24ddaf97 (127.9.77.130:40817): Not found: Tablet not found: 097ea812b3764b95bc781c17e6f2f808
I20260810 01:49:51.562266 466 leader_election.cc:302] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b4562ab42d4461e84c13b245eec5b5a; no voters: 4a02a22a13354c07bb0cde6f7c5b4d4e, 770f1e22d67044aa9525812b24ddaf97
I20260810 01:49:51.562651 1019 raft_consensus.cc:2726] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 01:49:52.009213 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:52.008845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.129:58271)
0810 01:49:52.009009 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:52.009086 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 01:49:52.212478 1025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:52.211906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.130:41405)
0810 01:49:52.212276 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:52.212344 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":150,"threads_started":1}
I20260810 01:49:52.437495 1029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:52.437024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.131:49033)
0810 01:49:52.437315 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:52.437377 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":101,"threads_started":1}
I20260810 01:49:53.010804 1030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:53.010238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.129:57377)
0810 01:49:53.010588 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:53.010662 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":106,"threads_started":1}
I20260810 01:49:53.214118 1032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:53.213543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.130:37857)
0810 01:49:53.213899 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:53.213981 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":104,"threads_started":1}
I20260810 01:49:53.439008 1033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:53.438459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.131:60269)
0810 01:49:53.438810 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:53.438878 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":138,"threads_started":1}
I20260810 01:49:53.670406 1034 raft_consensus.cc:482] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:49:53.670643 1034 raft_consensus.cc:504] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } }
I20260810 01:49:53.671281 1034 leader_election.cc:288] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a02a22a13354c07bb0cde6f7c5b4d4e (127.9.77.131:35807), 770f1e22d67044aa9525812b24ddaf97 (127.9.77.130:40817)
W20260810 01:49:53.672406 466 leader_election.cc:341] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 770f1e22d67044aa9525812b24ddaf97 (127.9.77.130:40817): Not found: Tablet not found: 097ea812b3764b95bc781c17e6f2f808
W20260810 01:49:53.672566 459 leader_election.cc:341] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4a02a22a13354c07bb0cde6f7c5b4d4e (127.9.77.131:35807): Not found: Tablet not found: 097ea812b3764b95bc781c17e6f2f808
I20260810 01:49:53.672801 459 leader_election.cc:302] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b4562ab42d4461e84c13b245eec5b5a; no voters: 4a02a22a13354c07bb0cde6f7c5b4d4e, 770f1e22d67044aa9525812b24ddaf97
I20260810 01:49:53.672993 1034 raft_consensus.cc:2726] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 01:49:54.012363 1035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:54.011761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.129:34903)
0810 01:49:54.012157 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:54.012227 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":105,"threads_started":1}
I20260810 01:49:54.215590 1039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:54.215089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.130:40163)
0810 01:49:54.215399 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:54.215468 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":105,"threads_started":1}
I20260810 01:49:54.440527 1042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:54.439944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.131:45107)
0810 01:49:54.440340 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:54.440404 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":164,"threads_started":1}
I20260810 01:49:55.014202 1044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:55.013576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.129:33867)
0810 01:49:55.013997 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:55.014069 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":152,"threads_started":1}
I20260810 01:49:55.217329 1045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:55.216767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.130:51277)
0810 01:49:55.217117 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:55.217193 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":107,"threads_started":1}
I20260810 01:49:55.281082 1046 raft_consensus.cc:482] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:49:55.281364 1046 raft_consensus.cc:504] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } }
I20260810 01:49:55.282023 1046 leader_election.cc:288] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a02a22a13354c07bb0cde6f7c5b4d4e (127.9.77.131:35807), 770f1e22d67044aa9525812b24ddaf97 (127.9.77.130:40817)
W20260810 01:49:55.283138 466 leader_election.cc:341] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 770f1e22d67044aa9525812b24ddaf97 (127.9.77.130:40817): Not found: Tablet not found: 097ea812b3764b95bc781c17e6f2f808
W20260810 01:49:55.283473 459 leader_election.cc:341] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4a02a22a13354c07bb0cde6f7c5b4d4e (127.9.77.131:35807): Not found: Tablet not found: 097ea812b3764b95bc781c17e6f2f808
I20260810 01:49:55.283586 459 leader_election.cc:302] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b4562ab42d4461e84c13b245eec5b5a; no voters: 4a02a22a13354c07bb0cde6f7c5b4d4e, 770f1e22d67044aa9525812b24ddaf97
I20260810 01:49:55.283780 1046 raft_consensus.cc:2726] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 01:49:55.442265 1047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:55.441611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.131:43215)
0810 01:49:55.442038 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:55.442118 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":135,"threads_started":1}
I20260810 01:49:56.016003 1048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:56.015317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.129:53027)
0810 01:49:56.015757 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:56.015852 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":127,"threads_started":1}
I20260810 01:49:56.219151 1049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:56.218537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.130:33387)
0810 01:49:56.218924 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:56.219006 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":113,"threads_started":1}
I20260810 01:49:56.443962 1052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:56.443372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.131:45083)
0810 01:49:56.443741 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:56.443821 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":137,"threads_started":1}
I20260810 01:49:57.017726 1053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:57.017155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.129:34883)
0810 01:49:57.017515 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:57.017583 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":144,"threads_started":1}
I20260810 01:49:57.220912 1054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:57.220248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.130:39599)
0810 01:49:57.220702 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:57.220769 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":153,"threads_started":1}
I20260810 01:49:57.445642 1055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:57.445015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.131:47743)
0810 01:49:57.445439 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:57.445510 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":103,"threads_started":1}
I20260810 01:49:58.019251 1057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:58.018659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.129:48769)
0810 01:49:58.019041 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:58.019113 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":115,"threads_started":1}
I20260810 01:49:58.222621 1058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:58.222025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.130:46531)
0810 01:49:58.222401 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:58.222481 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":135,"threads_started":1}
I20260810 01:49:58.447284 1059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:58.446717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.131:49375)
0810 01:49:58.447078 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:58.447151 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":138,"threads_started":1}
I20260810 01:49:59.020844 1060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:59.020320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.129:46725)
0810 01:49:59.020641 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:59.020709 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":106,"threads_started":1}
I20260810 01:49:59.044134 1063 raft_consensus.cc:482] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:49:59.044358 1063 raft_consensus.cc:504] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a02a22a13354c07bb0cde6f7c5b4d4e" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35807 } } peers { permanent_uuid: "770f1e22d67044aa9525812b24ddaf97" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40817 } } peers { permanent_uuid: "1b4562ab42d4461e84c13b245eec5b5a" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 45863 } }
I20260810 01:49:59.044943 1063 leader_election.cc:288] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a02a22a13354c07bb0cde6f7c5b4d4e (127.9.77.131:35807), 770f1e22d67044aa9525812b24ddaf97 (127.9.77.130:40817)
W20260810 01:49:59.045905 466 leader_election.cc:341] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 770f1e22d67044aa9525812b24ddaf97 (127.9.77.130:40817): Not found: Tablet not found: 097ea812b3764b95bc781c17e6f2f808
W20260810 01:49:59.046070 459 leader_election.cc:341] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4a02a22a13354c07bb0cde6f7c5b4d4e (127.9.77.131:35807): Not found: Tablet not found: 097ea812b3764b95bc781c17e6f2f808
I20260810 01:49:59.046168 459 leader_election.cc:302] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b4562ab42d4461e84c13b245eec5b5a; no voters: 4a02a22a13354c07bb0cde6f7c5b4d4e, 770f1e22d67044aa9525812b24ddaf97
I20260810 01:49:59.046367 1063 raft_consensus.cc:2726] T 097ea812b3764b95bc781c17e6f2f808 P 1b4562ab42d4461e84c13b245eec5b5a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 01:49:59.224367 1064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:59.223650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.130:40297)
0810 01:49:59.224047 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:59.224147 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":105,"threads_started":1}
I20260810 01:49:59.448961 1066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:49:59.448419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.131:60649)
0810 01:49:59.448760 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0810 01:49:59.448826 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":112,"threads_started":1}
I20260810 01:50:00.022702 1067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:00.022127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.129:32871)
0810 01:50:00.022478 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:00.022554 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":108,"threads_started":1}
I20260810 01:50:00.226182 1070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:00.225583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.130:36017)
0810 01:50:00.225952 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:00.226038 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":103,"threads_started":1}
I20260810 01:50:00.450815 1071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:00.450181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.131:54409)
0810 01:50:00.450591 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:00.450674 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":120,"threads_started":1}
I20260810 01:50:01.024608 1072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:01.023884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.129:37003)
0810 01:50:01.024326 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:01.024407 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":183,"threads_started":1}
I20260810 01:50:01.228089 1075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:01.227404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.130:43053)
0810 01:50:01.227802 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:01.227884 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":118,"threads_started":1}
I20260810 01:50:01.452483 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:01.451908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.131:33515)
0810 01:50:01.452273 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:01.452351 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":167,"threads_started":1}
I20260810 01:50:02.026286 1077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:02.025743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.129:36007)
0810 01:50:02.026076 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:02.026150 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":103,"threads_started":1}
I20260810 01:50:02.229890 1080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:02.229224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.130:34005)
0810 01:50:02.229661 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:02.229741 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":140,"threads_started":1}
I20260810 01:50:02.453944 1081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:02.453420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.131:40995)
0810 01:50:02.453743 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:02.453814 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":106,"threads_started":1}
I20260810 01:50:03.027978 1082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:03.027330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.129:35141)
0810 01:50:03.027724 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:03.027802 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":134,"threads_started":1}
I20260810 01:50:03.231483 1084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:03.230883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.130:38541)
0810 01:50:03.231249 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:03.231317 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":119,"threads_started":1}
I20260810 01:50:03.455902 1085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:03.455063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.131:36859)
0810 01:50:03.455513 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:03.455578 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":160,"threads_started":1}
I20260810 01:50:04.029651 1086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:04.029079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.129:58807)
0810 01:50:04.029434 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:04.029509 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":102,"threads_started":1}
I20260810 01:50:04.233263 1088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:04.232662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.130:60861)
0810 01:50:04.233038 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:04.233118 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":133,"threads_started":1}
I20260810 01:50:04.457568 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:04.457003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.131:57965)
0810 01:50:04.457356 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:04.457429 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":112,"threads_started":1}
I20260810 01:50:05.031486 1090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:05.030931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.129:57163)
0810 01:50:05.031270 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:05.031341 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":96,"threads_started":1}
I20260810 01:50:05.234980 1091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:05.234365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.130:48551)
0810 01:50:05.234782 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:05.234850 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":110,"threads_started":1}
I20260810 01:50:05.437618 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 382
I20260810 01:50:05.459700 1092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:05.459044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:46111 (local address 127.9.77.131:35667)
0810 01:50:05.459480 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:05.459558 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:46111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":120,"threads_started":1}
I20260810 01:50:05.460775 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 632
I20260810 01:50:05.477680 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 822
2026-08-10T01:50:05Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (20369 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:50:05 dist-test-slave-kvfs krb5kdc[1098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:50:05 dist-test-slave-kvfs krb5kdc[1098](info): set up 2 sockets
Aug 10 01:50:05 dist-test-slave-kvfs krb5kdc[1098](info): commencing operation
krb5kdc: starting...
W20260810 01:50:06.772912 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.265s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:50:06 dist-test-slave-kvfs krb5kdc[1098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:50:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:50:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:50:06.923909 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:37329
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:37329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:50:07.063933 1117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:50:07.064376 1117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:50:07.064460 1117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:50:07.068682 1117 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 01:50:07.068799 1117 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 01:50:07.068825 1117 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 01:50:07.068849 1117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:50:07.068892 1117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:50:07.068939 1117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:50:07.068975 1117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:50:07.069012 1117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:50:07.074908 1117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:37329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:37329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:50:07.076462 1117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:50:07.077832 1117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:50:07.088752 1129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:50:07.088788 1117 server_base.cc:1034] running on GCE node
W20260810 01:50:07.088752 1127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:50:07.089035 1131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:50:07.089783 1117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:50:07.091195 1117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:50:07.092371 1117 hybrid_clock.cc:648] HybridClock initialized: now 1786326607092355 us; error 39 us; skew 500 ppm
Aug 10 01:50:07 dist-test-slave-kvfs krb5kdc[1098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326607, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:50:07.095894 1117 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:50:07.097084 1117 webserver.cc:460] Webserver started at http://127.9.77.190:41769/ using document root <none> and password file <none>
I20260810 01:50:07.097808 1117 fs_manager.cc:359] Metadata directory not provided
I20260810 01:50:07.097909 1117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:50:07.098181 1117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:50:07.100826 1117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf"
format_stamp: "Formatted at 2026-08-10 01:50:07 on dist-test-slave-kvfs"
server_key: "9ae0e0b60e96a77c64c6c6a2d8649746"
server_key_iv: "7bd16d3bc162ade345c5df42b9f04ac8"
server_key_version: "encryptionkey@0"
I20260810 01:50:07.101502 1117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf"
format_stamp: "Formatted at 2026-08-10 01:50:07 on dist-test-slave-kvfs"
server_key: "9ae0e0b60e96a77c64c6c6a2d8649746"
server_key_iv: "7bd16d3bc162ade345c5df42b9f04ac8"
server_key_version: "encryptionkey@0"
I20260810 01:50:07.106288 1117 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 01:50:07.109450 1145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:50:07.111064 1117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:50:07.111260 1117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf"
format_stamp: "Formatted at 2026-08-10 01:50:07 on dist-test-slave-kvfs"
server_key: "9ae0e0b60e96a77c64c6c6a2d8649746"
server_key_iv: "7bd16d3bc162ade345c5df42b9f04ac8"
server_key_version: "encryptionkey@0"
I20260810 01:50:07.111404 1117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:50:07.123966 1117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:50:07.128129 1117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:50:07.128383 1117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:50:07.137717 1117 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:37329
I20260810 01:50:07.138139 1213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:37329 every 8 connection(s)
I20260810 01:50:07.139034 1117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:50:07.142766 1214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:50:07.144479 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 1117
I20260810 01:50:07.144656 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:50:07.145059 9526 external_mini_cluster.cc:1428] Setting key b0caca9c24bc8d564eecec88f24ebd6c
I20260810 01:50:07.150882 1214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf: Bootstrap starting.
I20260810 01:50:07.154279 1214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf: Neither blocks nor log segments found. Creating new log.
I20260810 01:50:07.155450 1214 log.cc:824] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf: Log is configured to *not* fsync() on all Append() calls
Aug 10 01:50:07 dist-test-slave-kvfs krb5kdc[1098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:50:07.158026 1214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf: No bootstrap required, opened a new log
I20260810 01:50:07.162209 1219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:07.146844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34626 (local address 127.9.77.190:37329)
0810 01:50:07.147385 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0810 01:50:07.147406 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 01:50:07.147466 (+ 60us) server_negotiation.cc:407] Connection header received
0810 01:50:07.148332 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:50:07.148355 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:50:07.148774 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:50:07.149110 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:50:07.150244 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:07.151399 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:07.152648 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:07.153178 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:07.153322 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:07.156779 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:50:07.156785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:50:07.156802 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:50:07.156834 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:50:07.158896 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:50:07.159370 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:50:07.159376 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:50:07.159389 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:50:07.159520 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:50:07.159825 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:50:07.159829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:50:07.159831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:50:07.160403 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:50:07.160654 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0810 01:50:07.160939 (+ 285us) server_negotiation.cc:299] Negotiation successful
0810 01:50:07.161294 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":173,"threads_started":1}
I20260810 01:50:07.162667 1214 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37329 } }
I20260810 01:50:07.162966 1214 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:50:07.163064 1214 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f046c3d7cc5420f9018b43ef4c2c6bf, State: Initialized, Role: FOLLOWER
I20260810 01:50:07.163574 1214 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37329 } }
I20260810 01:50:07.163722 1214 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:50:07.163820 1214 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:50:07.163930 1214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:50:07.165175 1214 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37329 } }
I20260810 01:50:07.165601 1214 leader_election.cc:302] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f046c3d7cc5420f9018b43ef4c2c6bf; no voters:
I20260810 01:50:07.166000 1214 leader_election.cc:288] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:50:07.166145 1221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:50:07.166448 1221 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [term 1 LEADER]: Becoming Leader. State: Replica: 2f046c3d7cc5420f9018b43ef4c2c6bf, State: Running, Role: LEADER
I20260810 01:50:07.166857 1221 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37329 } }
I20260810 01:50:07.167539 1214 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:50:07.169054 1223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f046c3d7cc5420f9018b43ef4c2c6bf. Latest consensus state: current_term: 1 leader_uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37329 } } }
I20260810 01:50:07.169220 1223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [sys.catalog]: This master's current role is: LEADER
I20260810 01:50:07.169669 1233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:50:07.169780 1222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37329 } } }
I20260810 01:50:07.170076 1222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [sys.catalog]: This master's current role is: LEADER
I20260810 01:50:07.173447 1233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:50:07.180332 1233 catalog_manager.cc:1347] Generated new cluster ID: e87b153867594db6ae349e9b0ad636ad
I20260810 01:50:07.180435 1233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:50:07.191371 1233 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:50:07.192847 1233 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:50:07.204927 1233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf: Generated new TSK 0
I20260810 01:50:07.205927 1233 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.77.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
I20260810 01:50:07.244570 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:0
--local_ip_for_outbound_sockets=127.9.77.129
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:37329
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:50:07.392210 1254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:50:07.392540 1254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:50:07.392590 1254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:50:07.396901 1254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:50:07.396979 1254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:50:07.397044 1254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:50:07.402818 1254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:37329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:50:07.404407 1254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:50:07.405817 1254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:50:07.417604 1264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:50:07.417604 1266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:50:07.417627 1263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:50:07.418061 1254 server_base.cc:1034] running on GCE node
I20260810 01:50:07.418597 1254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:50:07.419401 1254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:50:07.420619 1254 hybrid_clock.cc:648] HybridClock initialized: now 1786326607420586 us; error 45 us; skew 500 ppm
Aug 10 01:50:07 dist-test-slave-kvfs krb5kdc[1098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326607, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:50:07.424126 1254 init.cc:377] Logged in from keytab as kudu/127.9.77.129@KRBTEST.COM (short username kudu)
I20260810 01:50:07.425562 1254 webserver.cc:460] Webserver started at http://127.9.77.129:36209/ using document root <none> and password file <none>
I20260810 01:50:07.426276 1254 fs_manager.cc:359] Metadata directory not provided
I20260810 01:50:07.426383 1254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:50:07.426671 1254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:50:07.429476 1254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/data/instance:
uuid: "332d12f203e04e69aadf05e9c8ad000e"
format_stamp: "Formatted at 2026-08-10 01:50:07 on dist-test-slave-kvfs"
server_key: "2a7d1f09c957083b42f2ee305e3a12c0"
server_key_iv: "6472c5cb9e4fc1b72a69ec99b3129422"
server_key_version: "encryptionkey@0"
I20260810 01:50:07.430150 1254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance:
uuid: "332d12f203e04e69aadf05e9c8ad000e"
format_stamp: "Formatted at 2026-08-10 01:50:07 on dist-test-slave-kvfs"
server_key: "2a7d1f09c957083b42f2ee305e3a12c0"
server_key_iv: "6472c5cb9e4fc1b72a69ec99b3129422"
server_key_version: "encryptionkey@0"
I20260810 01:50:07.434723 1254 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 01:50:07.437708 1275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:50:07.439677 1254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 01:50:07.439814 1254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "332d12f203e04e69aadf05e9c8ad000e"
format_stamp: "Formatted at 2026-08-10 01:50:07 on dist-test-slave-kvfs"
server_key: "2a7d1f09c957083b42f2ee305e3a12c0"
server_key_iv: "6472c5cb9e4fc1b72a69ec99b3129422"
server_key_version: "encryptionkey@0"
I20260810 01:50:07.439936 1254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:50:07.461589 1254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:50:07.465515 1254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:50:07.465765 1254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:50:07.466615 1254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:50:07.467730 1254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:50:07.467780 1254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:50:07.467869 1254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:50:07.467901 1254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:50:07.478873 1254 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:32833
I20260810 01:50:07.478899 1455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:32833 every 8 connection(s)
I20260810 01:50:07.480062 1254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:50:07.486635 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 1254
I20260810 01:50:07.486809 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance
I20260810 01:50:07.487174 9526 external_mini_cluster.cc:1428] Setting key 00573523e37d221168d8c41a741038ea
Aug 10 01:50:07 dist-test-slave-kvfs krb5kdc[1098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326607, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:50:07.497722 1219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:07.482426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:34631 (local address 127.9.77.190:37329)
0810 01:50:07.482596 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 01:50:07.482600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:50:07.483541 (+ 941us) server_negotiation.cc:407] Connection header received
0810 01:50:07.485392 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:50:07.485397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:50:07.485475 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:50:07.485595 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:50:07.487160 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:07.487988 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:07.489675 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:07.489938 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:07.490023 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:07.493668 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:50:07.493672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:50:07.493675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:50:07.493703 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:50:07.495201 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:50:07.495759 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:50:07.495763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:50:07.495770 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:50:07.495853 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:50:07.496363 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:50:07.496365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:50:07.496366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:50:07.496568 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:50:07.496652 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 01:50:07.497405 (+ 753us) server_negotiation.cc:299] Negotiation successful
0810 01:50:07.497533 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 01:50:07.499239 1458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:07.482823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.129:34631)
0810 01:50:07.483360 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:07.483410 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 01:50:07.485128 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:50:07.485809 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:50:07.485817 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:50:07.486259 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:50:07.486986 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:07.487005 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:07.488185 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:07.488188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:07.489520 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:07.489534 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:07.490110 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:07.490116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:07.490391 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:07.491182 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:50:07.491212 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:50:07.493473 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:50:07.495326 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:50:07.495332 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:50:07.495351 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:50:07.495642 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:50:07.496001 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:50:07.496004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:50:07.496006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:50:07.496250 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:50:07.496687 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:50:07.496696 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:50:07.497100 (+ 404us) client_negotiation.cc:769] Sending connection context
0810 01:50:07.497372 (+ 272us) client_negotiation.cc:240] Negotiation successful
0810 01:50:07.497854 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":135,"threads_started":1}
I20260810 01:50:07.500828 1456 heartbeater.cc:347] Connected to a master server at 127.9.77.190:37329
WARNING: no policy specified for kudu/127.9.77.130@KRBTEST.COM; defaulting to no policy
I20260810 01:50:07.501142 1456 heartbeater.cc:464] Registering TS with master...
I20260810 01:50:07.501766 1456 heartbeater.cc:511] Master 127.9.77.190:37329 requested a full tablet report, sending...
I20260810 01:50:07.503849 1166 ts_manager.cc:194] Registered new tserver with Master: 332d12f203e04e69aadf05e9c8ad000e (127.9.77.129:32833)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.130@KRBTEST.COM" created.
I20260810 01:50:07.505766 1166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.129@KRBTEST.COM'} at 127.9.77.129:34631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
I20260810 01:50:07.524845 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:0
--local_ip_for_outbound_sockets=127.9.77.130
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:37329
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:50:07.658738 1461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:50:07.659132 1461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:50:07.659219 1461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:50:07.663321 1461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:50:07.663426 1461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:50:07.663527 1461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:50:07.669332 1461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:37329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:50:07.670711 1461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:50:07.672048 1461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:50:07.684120 1477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:50:07.684120 1469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:50:07.684167 1467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:50:07.684661 1461 server_base.cc:1034] running on GCE node
I20260810 01:50:07.685293 1461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:50:07.686115 1461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:50:07.687297 1461 hybrid_clock.cc:648] HybridClock initialized: now 1786326607687276 us; error 34 us; skew 500 ppm
Aug 10 01:50:07 dist-test-slave-kvfs krb5kdc[1098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326607, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:50:07.690829 1461 init.cc:377] Logged in from keytab as kudu/127.9.77.130@KRBTEST.COM (short username kudu)
I20260810 01:50:07.692178 1461 webserver.cc:460] Webserver started at http://127.9.77.130:33415/ using document root <none> and password file <none>
I20260810 01:50:07.692973 1461 fs_manager.cc:359] Metadata directory not provided
I20260810 01:50:07.693090 1461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:50:07.693413 1461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:50:07.696223 1461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/data/instance:
uuid: "5ed840bbf34b4186b6778095329ee077"
format_stamp: "Formatted at 2026-08-10 01:50:07 on dist-test-slave-kvfs"
server_key: "beecc5c2b6695de93ac8a42f736ba0f8"
server_key_iv: "6e32b801a8b77c04597e7bb0dd1413e0"
server_key_version: "encryptionkey@0"
I20260810 01:50:07.696969 1461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance:
uuid: "5ed840bbf34b4186b6778095329ee077"
format_stamp: "Formatted at 2026-08-10 01:50:07 on dist-test-slave-kvfs"
server_key: "beecc5c2b6695de93ac8a42f736ba0f8"
server_key_iv: "6e32b801a8b77c04597e7bb0dd1413e0"
server_key_version: "encryptionkey@0"
I20260810 01:50:07.701938 1461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 01:50:07.705005 1489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:50:07.706571 1461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:50:07.706688 1461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "5ed840bbf34b4186b6778095329ee077"
format_stamp: "Formatted at 2026-08-10 01:50:07 on dist-test-slave-kvfs"
server_key: "beecc5c2b6695de93ac8a42f736ba0f8"
server_key_iv: "6e32b801a8b77c04597e7bb0dd1413e0"
server_key_version: "encryptionkey@0"
I20260810 01:50:07.706871 1461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:50:07.734225 1461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:50:07.738122 1461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:50:07.738387 1461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:50:07.739202 1461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:50:07.740309 1461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:50:07.740361 1461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:50:07.740437 1461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:50:07.740471 1461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:50:07.751456 1461 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:38569
I20260810 01:50:07.751507 1660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:38569 every 8 connection(s)
I20260810 01:50:07.752722 1461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:50:07.754995 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 1461
I20260810 01:50:07.755152 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance
I20260810 01:50:07.755499 9526 external_mini_cluster.cc:1428] Setting key 94c6efe89c4377c310e28e0559418ad2
Aug 10 01:50:07 dist-test-slave-kvfs krb5kdc[1098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326607, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.77.131@KRBTEST.COM; defaulting to no policy
I20260810 01:50:07.770188 1219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:07.755775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:38399 (local address 127.9.77.190:37329)
0810 01:50:07.755993 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 01:50:07.756000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:50:07.757432 (+ 1432us) server_negotiation.cc:407] Connection header received
0810 01:50:07.758602 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:50:07.758608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:50:07.758690 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:50:07.758803 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:50:07.760404 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:07.761004 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:07.762319 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:07.762583 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:07.762710 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:07.765831 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:50:07.765835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:50:07.765837 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:50:07.765870 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:50:07.767483 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:50:07.768169 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:50:07.768174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:50:07.768176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:50:07.768269 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:50:07.768660 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:50:07.768663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:50:07.768665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:50:07.769022 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:50:07.769142 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 01:50:07.769810 (+ 668us) server_negotiation.cc:299] Negotiation successful
0810 01:50:07.769977 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 01:50:07.771407 1663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:07.756009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.130:38399)
0810 01:50:07.757255 (+ 1246us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:07.757321 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 01:50:07.758346 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:50:07.759039 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:50:07.759048 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:50:07.759504 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:50:07.760228 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:07.760247 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:07.761151 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:07.761155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:07.762174 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:07.762184 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:07.762761 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:07.762765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:07.762980 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:07.763791 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:50:07.763810 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:50:07.765624 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:50:07.767616 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:50:07.767623 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:50:07.767644 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:50:07.767990 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:50:07.768367 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:50:07.768372 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:50:07.768377 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:50:07.768551 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:50:07.769137 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:50:07.769146 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:50:07.769482 (+ 336us) client_negotiation.cc:769] Sending connection context
0810 01:50:07.769707 (+ 225us) client_negotiation.cc:240] Negotiation successful
0810 01:50:07.770188 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"thread_start_us":756,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.131@KRBTEST.COM" created.
I20260810 01:50:07.773015 1661 heartbeater.cc:347] Connected to a master server at 127.9.77.190:37329
I20260810 01:50:07.773365 1661 heartbeater.cc:464] Registering TS with master...
I20260810 01:50:07.774132 1661 heartbeater.cc:511] Master 127.9.77.190:37329 requested a full tablet report, sending...
I20260810 01:50:07.775722 1166 ts_manager.cc:194] Registered new tserver with Master: 5ed840bbf34b4186b6778095329ee077 (127.9.77.130:38569)
I20260810 01:50:07.776796 1166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.130@KRBTEST.COM'} at 127.9.77.130:38399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
I20260810 01:50:07.793201 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:0
--local_ip_for_outbound_sockets=127.9.77.131
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:37329
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:50:07.933672 1669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:50:07.934052 1669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:50:07.934105 1669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:50:07.938256 1669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:50:07.938318 1669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:50:07.938381 1669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:50:07.944144 1669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:37329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:50:07.945598 1669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:50:07.946883 1669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:50:07.958885 1676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:50:07.958890 1678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:50:07.958907 1675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:50:07.958949 1669 server_base.cc:1034] running on GCE node
I20260810 01:50:07.959617 1669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:50:07.960446 1669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:50:07.961635 1669 hybrid_clock.cc:648] HybridClock initialized: now 1786326607961619 us; error 28 us; skew 500 ppm
Aug 10 01:50:07 dist-test-slave-kvfs krb5kdc[1098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326607, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:50:07.965206 1669 init.cc:377] Logged in from keytab as kudu/127.9.77.131@KRBTEST.COM (short username kudu)
I20260810 01:50:07.966290 1669 webserver.cc:460] Webserver started at http://127.9.77.131:43823/ using document root <none> and password file <none>
I20260810 01:50:07.967008 1669 fs_manager.cc:359] Metadata directory not provided
I20260810 01:50:07.967077 1669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:50:07.967355 1669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:50:07.969976 1669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/data/instance:
uuid: "3c2f96c5d7a74c168d0489da8797f3e1"
format_stamp: "Formatted at 2026-08-10 01:50:07 on dist-test-slave-kvfs"
server_key: "20b51934f150a65e74b3399f11571ce5"
server_key_iv: "d804fdf07a789dbeb2c48a7e617a75e8"
server_key_version: "encryptionkey@0"
I20260810 01:50:07.970676 1669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance:
uuid: "3c2f96c5d7a74c168d0489da8797f3e1"
format_stamp: "Formatted at 2026-08-10 01:50:07 on dist-test-slave-kvfs"
server_key: "20b51934f150a65e74b3399f11571ce5"
server_key_iv: "d804fdf07a789dbeb2c48a7e617a75e8"
server_key_version: "encryptionkey@0"
I20260810 01:50:07.975327 1669 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 01:50:07.978451 1688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:50:07.979961 1669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 01:50:07.980170 1669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "3c2f96c5d7a74c168d0489da8797f3e1"
format_stamp: "Formatted at 2026-08-10 01:50:07 on dist-test-slave-kvfs"
server_key: "20b51934f150a65e74b3399f11571ce5"
server_key_iv: "d804fdf07a789dbeb2c48a7e617a75e8"
server_key_version: "encryptionkey@0"
I20260810 01:50:07.980314 1669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:50:07.997385 1669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:50:08.002130 1669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:50:08.002388 1669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:50:08.003257 1669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:50:08.004346 1669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:50:08.004400 1669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:50:08.004451 1669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:50:08.004472 1669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:50:08.015244 1669 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:46805
I20260810 01:50:08.015269 1859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:46805 every 8 connection(s)
I20260810 01:50:08.016644 1669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:50:08.023511 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 1669
I20260810 01:50:08.023692 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance
I20260810 01:50:08.024098 9526 external_mini_cluster.cc:1428] Setting key 0a9f331edb7a8c745e9913b53b7d36cf
Aug 10 01:50:08 dist-test-slave-kvfs krb5kdc[1098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326607, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:50:08.034312 1219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:08.019019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:58305 (local address 127.9.77.190:37329)
0810 01:50:08.019224 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 01:50:08.019230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:50:08.020346 (+ 1116us) server_negotiation.cc:407] Connection header received
0810 01:50:08.021902 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:50:08.021905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:50:08.021966 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:50:08.022051 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:50:08.023488 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:08.024402 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:08.026046 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:08.026397 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:08.026513 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:08.029915 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:50:08.029919 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:50:08.029922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:50:08.029956 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:50:08.031824 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:50:08.032525 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:50:08.032529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:50:08.032531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:50:08.032635 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:50:08.033016 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:50:08.033019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:50:08.033020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:50:08.033253 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:50:08.033337 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 01:50:08.033963 (+ 626us) server_negotiation.cc:299] Negotiation successful
0810 01:50:08.034151 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 01:50:08.035378 1865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:08.019554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.131:58305)
0810 01:50:08.020171 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:08.020224 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 01:50:08.021673 (+ 1449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:50:08.022255 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:50:08.022262 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:50:08.022714 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:50:08.023316 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:08.023331 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:08.024592 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:08.024596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:08.025926 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:08.025943 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:08.026714 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:08.026717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:08.026969 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:08.027651 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:50:08.027671 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:50:08.029655 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:50:08.031975 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:50:08.031982 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:50:08.032001 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:50:08.032395 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:50:08.032733 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:50:08.032735 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:50:08.032737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:50:08.032915 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:50:08.033347 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:50:08.033352 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:50:08.033663 (+ 311us) client_negotiation.cc:769] Sending connection context
0810 01:50:08.033852 (+ 189us) client_negotiation.cc:240] Negotiation successful
0810 01:50:08.034216 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":129,"threads_started":1}
I20260810 01:50:08.036852 1860 heartbeater.cc:347] Connected to a master server at 127.9.77.190:37329
I20260810 01:50:08.037191 1860 heartbeater.cc:464] Registering TS with master...
I20260810 01:50:08.038023 1860 heartbeater.cc:511] Master 127.9.77.190:37329 requested a full tablet report, sending...
I20260810 01:50:08.039381 1166 ts_manager.cc:194] Registered new tserver with Master: 3c2f96c5d7a74c168d0489da8797f3e1 (127.9.77.131:46805)
I20260810 01:50:08.040104 1166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.131@KRBTEST.COM'} at 127.9.77.131:58305
I20260810 01:50:08.051352 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 01:50:08.062282 1219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:08.054029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34640 (local address 127.9.77.190:37329)
0810 01:50:08.054181 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 01:50:08.054186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:50:08.054325 (+ 139us) server_negotiation.cc:407] Connection header received
0810 01:50:08.054450 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:50:08.054452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:50:08.054504 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:50:08.054581 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:50:08.055417 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:08.056174 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:08.057097 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:08.057337 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:08.057431 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:08.058974 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:50:08.058977 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:50:08.058979 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:50:08.059000 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:50:08.060680 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:50:08.061082 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:50:08.061085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:50:08.061086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:50:08.061171 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:50:08.061409 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:50:08.061411 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:50:08.061412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:50:08.061676 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:50:08.061797 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 01:50:08.062015 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 01:50:08.062129 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 01:50:08.066090 1166 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 01:50:08.070392 1166 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:50:08.088171 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:08.082172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:46805 (local address 127.0.0.1:37856)
0810 01:50:08.082711 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:08.082746 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 01:50:08.082846 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:50:08.083619 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:50:08.083622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:50:08.083651 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:50:08.083864 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:08.083870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:08.085341 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:08.085345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:08.086572 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:08.086582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:08.087476 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:08.087479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:08.087606 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:08.087653 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 01:50:08.087771 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 01:50:08.087924 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":107,"spinlock_wait_cycles":4864,"thread_start_us":129,"threads_started":1}
I20260810 01:50:08.088858 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:08.082446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37856 (local address 127.9.77.131:46805)
0810 01:50:08.082956 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0810 01:50:08.082972 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 01:50:08.082993 (+ 21us) server_negotiation.cc:407] Connection header received
0810 01:50:08.083065 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:50:08.083070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:50:08.083284 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:50:08.083452 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:50:08.084006 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:08.085200 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:08.086710 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:08.087330 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:08.087478 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:08.088503 (+ 1025us) server_negotiation.cc:1035] Waiting for connection context
0810 01:50:08.088566 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 01:50:08.088665 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":138,"threads_started":1}
I20260810 01:50:08.089323 1879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:08.081274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:38569 (local address 127.0.0.1:50040)
0810 01:50:08.082518 (+ 1244us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:08.082582 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 01:50:08.082964 (+ 382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:50:08.083462 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:50:08.083469 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:50:08.083494 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:50:08.083772 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:08.083781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:08.085708 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:08.085711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:08.087364 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:08.087376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:08.088837 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:08.088841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:08.088963 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:08.088991 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 01:50:08.089057 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 01:50:08.089129 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"mutex_wait_us":136,"spinlock_wait_cycles":569728,"thread_start_us":115,"threads_started":1}
I20260810 01:50:08.089572 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:08.081773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50040 (local address 127.9.77.130:38569)
0810 01:50:08.082964 (+ 1191us) server_negotiation.cc:206] Beginning negotiation
0810 01:50:08.082979 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 01:50:08.083013 (+ 34us) server_negotiation.cc:407] Connection header received
0810 01:50:08.083112 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:50:08.083116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:50:08.083282 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:50:08.083471 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:50:08.083954 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:08.085557 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:08.087545 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:08.088667 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:08.088813 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:08.089108 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0810 01:50:08.089172 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 01:50:08.089284 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1046,"thread_start_us":135,"threads_started":1}
I20260810 01:50:08.091502 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:08.081601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:32833 (local address 127.0.0.1:55186)
0810 01:50:08.082539 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:08.082595 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 01:50:08.082767 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:50:08.083619 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:50:08.083622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:50:08.083653 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:50:08.083847 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:08.083854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:08.085699 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:08.085703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:08.087364 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:08.087376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:08.091042 (+ 3666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:08.091047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:08.091147 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:08.091172 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 01:50:08.091262 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 01:50:08.091346 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"mutex_wait_us":221,"spinlock_wait_cycles":4352,"thread_start_us":125,"threads_started":1}
I20260810 01:50:08.092280 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:08.081830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55186 (local address 127.9.77.129:32833)
0810 01:50:08.082554 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0810 01:50:08.082565 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 01:50:08.082933 (+ 368us) server_negotiation.cc:407] Connection header received
0810 01:50:08.083044 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:50:08.083050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:50:08.083284 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:50:08.083452 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:50:08.083967 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:08.085534 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:08.087544 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:08.088803 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:08.088908 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:08.091896 (+ 2988us) server_negotiation.cc:1035] Waiting for connection context
0810 01:50:08.091960 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 01:50:08.092100 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":113,"threads_started":1}
I20260810 01:50:08.093206 1568 tablet_service.cc:1467] Processing CreateTablet for tablet a0003934666140618e1db53138901993 (DEFAULT_TABLE table=test-table [id=77e764c0b8054f84a5cf35e9a98ca0fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:50:08.093201 1761 tablet_service.cc:1467] Processing CreateTablet for tablet a0003934666140618e1db53138901993 (DEFAULT_TABLE table=test-table [id=77e764c0b8054f84a5cf35e9a98ca0fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:50:08.094404 1568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0003934666140618e1db53138901993. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:50:08.094422 1761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0003934666140618e1db53138901993. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:50:08.095361 1350 tablet_service.cc:1467] Processing CreateTablet for tablet a0003934666140618e1db53138901993 (DEFAULT_TABLE table=test-table [id=77e764c0b8054f84a5cf35e9a98ca0fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:50:08.096524 1350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0003934666140618e1db53138901993. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:50:08.101848 1885 tablet_bootstrap.cc:492] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077: Bootstrap starting.
I20260810 01:50:08.103499 1886 tablet_bootstrap.cc:492] T a0003934666140618e1db53138901993 P 3c2f96c5d7a74c168d0489da8797f3e1: Bootstrap starting.
I20260810 01:50:08.104897 1887 tablet_bootstrap.cc:492] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e: Bootstrap starting.
I20260810 01:50:08.105729 1886 tablet_bootstrap.cc:654] T a0003934666140618e1db53138901993 P 3c2f96c5d7a74c168d0489da8797f3e1: Neither blocks nor log segments found. Creating new log.
I20260810 01:50:08.106290 1885 tablet_bootstrap.cc:654] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077: Neither blocks nor log segments found. Creating new log.
I20260810 01:50:08.107369 1886 log.cc:824] T a0003934666140618e1db53138901993 P 3c2f96c5d7a74c168d0489da8797f3e1: Log is configured to *not* fsync() on all Append() calls
I20260810 01:50:08.107363 1885 log.cc:824] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077: Log is configured to *not* fsync() on all Append() calls
I20260810 01:50:08.107395 1887 tablet_bootstrap.cc:654] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e: Neither blocks nor log segments found. Creating new log.
I20260810 01:50:08.108511 1887 log.cc:824] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e: Log is configured to *not* fsync() on all Append() calls
I20260810 01:50:08.109957 1885 tablet_bootstrap.cc:492] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077: No bootstrap required, opened a new log
I20260810 01:50:08.109957 1886 tablet_bootstrap.cc:492] T a0003934666140618e1db53138901993 P 3c2f96c5d7a74c168d0489da8797f3e1: No bootstrap required, opened a new log
I20260810 01:50:08.110188 1885 ts_tablet_manager.cc:1397] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260810 01:50:08.110188 1886 ts_tablet_manager.cc:1397] T a0003934666140618e1db53138901993 P 3c2f96c5d7a74c168d0489da8797f3e1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 01:50:08.111104 1887 tablet_bootstrap.cc:492] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e: No bootstrap required, opened a new log
I20260810 01:50:08.111394 1887 ts_tablet_manager.cc:1397] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 01:50:08.114702 1886 raft_consensus.cc:348] T a0003934666140618e1db53138901993 P 3c2f96c5d7a74c168d0489da8797f3e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } }
I20260810 01:50:08.114702 1885 raft_consensus.cc:348] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } }
I20260810 01:50:08.115046 1885 raft_consensus.cc:374] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:50:08.115046 1886 raft_consensus.cc:374] T a0003934666140618e1db53138901993 P 3c2f96c5d7a74c168d0489da8797f3e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:50:08.115104 1885 raft_consensus.cc:729] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ed840bbf34b4186b6778095329ee077, State: Initialized, Role: FOLLOWER
I20260810 01:50:08.115104 1886 raft_consensus.cc:729] T a0003934666140618e1db53138901993 P 3c2f96c5d7a74c168d0489da8797f3e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c2f96c5d7a74c168d0489da8797f3e1, State: Initialized, Role: FOLLOWER
I20260810 01:50:08.115666 1885 consensus_queue.cc:260] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } }
I20260810 01:50:08.115666 1886 consensus_queue.cc:260] T a0003934666140618e1db53138901993 P 3c2f96c5d7a74c168d0489da8797f3e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } }
I20260810 01:50:08.115715 1887 raft_consensus.cc:348] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } }
I20260810 01:50:08.116044 1887 raft_consensus.cc:374] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:50:08.116128 1887 raft_consensus.cc:729] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 332d12f203e04e69aadf05e9c8ad000e, State: Initialized, Role: FOLLOWER
I20260810 01:50:08.116606 1887 consensus_queue.cc:260] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } }
I20260810 01:50:08.116917 1886 ts_tablet_manager.cc:1428] T a0003934666140618e1db53138901993 P 3c2f96c5d7a74c168d0489da8797f3e1: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 01:50:08.116917 1885 ts_tablet_manager.cc:1428] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077: Time spent starting tablet: real 0.007s user 0.000s sys 0.008s
I20260810 01:50:08.117345 1661 heartbeater.cc:503] Master 127.9.77.190:37329 was elected leader, sending a full tablet report...
I20260810 01:50:08.117345 1860 heartbeater.cc:503] Master 127.9.77.190:37329 was elected leader, sending a full tablet report...
I20260810 01:50:08.117391 1456 heartbeater.cc:503] Master 127.9.77.190:37329 was elected leader, sending a full tablet report...
I20260810 01:50:08.117743 1887 ts_tablet_manager.cc:1428] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260810 01:50:08.146721 1209 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.9.77.190:37329
W20260810 01:50:08.231905 1457 tablet.cc:2369] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 01:50:08.254366 1662 tablet.cc:2369] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 01:50:08.268357 1861 tablet.cc:2369] T a0003934666140618e1db53138901993 P 3c2f96c5d7a74c168d0489da8797f3e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 01:50:08.385835 1893 raft_consensus.cc:482] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:50:08.386189 1893 raft_consensus.cc:504] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } }
I20260810 01:50:08.387729 1893 leader_election.cc:288] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 332d12f203e04e69aadf05e9c8ad000e (127.9.77.129:32833), 3c2f96c5d7a74c168d0489da8797f3e1 (127.9.77.131:46805)
I20260810 01:50:08.393931 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:08.388885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:57741 (local address 127.9.77.131:46805)
0810 01:50:08.389069 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 01:50:08.389077 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 01:50:08.389211 (+ 134us) server_negotiation.cc:407] Connection header received
0810 01:50:08.389324 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:50:08.389328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:50:08.389391 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:50:08.389464 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:50:08.389761 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:08.390529 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:08.392339 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:08.392995 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:08.393141 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:08.393206 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 01:50:08.393675 (+ 469us) server_negotiation.cc:299] Negotiation successful
0810 01:50:08.393768 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 01:50:08.393931 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:08.388359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:44255 (local address 127.9.77.129:32833)
0810 01:50:08.388579 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 01:50:08.388586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:50:08.388741 (+ 155us) server_negotiation.cc:407] Connection header received
0810 01:50:08.388970 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:50:08.388974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:50:08.389046 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:50:08.389152 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:50:08.389545 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:08.390425 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:08.392334 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:08.392998 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:08.393141 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:08.393206 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 01:50:08.393670 (+ 464us) server_negotiation.cc:299] Negotiation successful
0810 01:50:08.393799 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 01:50:08.394001 1896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:08.388143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:32833 (local address 127.9.77.130:44255)
0810 01:50:08.388590 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:08.388610 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 01:50:08.388823 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:50:08.389187 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:50:08.389190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:50:08.389218 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:50:08.389396 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:08.389402 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:08.390562 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:08.390565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:08.392162 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:08.392173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:08.393171 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:08.393175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:08.393396 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:08.393472 (+ 76us) client_negotiation.cc:769] Sending connection context
0810 01:50:08.393574 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 01:50:08.393785 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":224,"spinlock_wait_cycles":6656,"thread_start_us":145,"threads_started":1}
I20260810 01:50:08.394001 1899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:08.388694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:46805 (local address 127.9.77.130:57741)
0810 01:50:08.389057 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:08.389069 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 01:50:08.389217 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:50:08.389510 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:50:08.389516 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:50:08.389538 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:50:08.389645 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:08.389649 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:08.390647 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:08.390650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:08.392163 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:08.392173 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:08.393171 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:08.393175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:08.393461 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:08.393486 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 01:50:08.393574 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 01:50:08.393773 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":238,"spinlock_wait_cycles":384,"thread_start_us":120,"threads_started":1}
I20260810 01:50:08.394958 1382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0003934666140618e1db53138901993" candidate_uuid: "5ed840bbf34b4186b6778095329ee077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "332d12f203e04e69aadf05e9c8ad000e" is_pre_election: true
I20260810 01:50:08.395455 1382 raft_consensus.cc:2454] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ed840bbf34b4186b6778095329ee077 in term 0.
I20260810 01:50:08.395516 1794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0003934666140618e1db53138901993" candidate_uuid: "5ed840bbf34b4186b6778095329ee077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" is_pre_election: true
I20260810 01:50:08.395828 1794 raft_consensus.cc:2454] T a0003934666140618e1db53138901993 P 3c2f96c5d7a74c168d0489da8797f3e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ed840bbf34b4186b6778095329ee077 in term 0.
I20260810 01:50:08.396153 1496 leader_election.cc:302] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 332d12f203e04e69aadf05e9c8ad000e, 5ed840bbf34b4186b6778095329ee077; no voters:
I20260810 01:50:08.396517 1893 raft_consensus.cc:2781] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:50:08.396607 1893 raft_consensus.cc:482] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:50:08.396638 1893 raft_consensus.cc:3037] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:50:08.397989 1893 raft_consensus.cc:504] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } }
I20260810 01:50:08.398406 1893 leader_election.cc:288] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077 [CANDIDATE]: Term 1 election: Requested vote from peers 332d12f203e04e69aadf05e9c8ad000e (127.9.77.129:32833), 3c2f96c5d7a74c168d0489da8797f3e1 (127.9.77.131:46805)
I20260810 01:50:08.398972 1382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0003934666140618e1db53138901993" candidate_uuid: "5ed840bbf34b4186b6778095329ee077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "332d12f203e04e69aadf05e9c8ad000e"
I20260810 01:50:08.399053 1794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0003934666140618e1db53138901993" candidate_uuid: "5ed840bbf34b4186b6778095329ee077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c2f96c5d7a74c168d0489da8797f3e1"
I20260810 01:50:08.399118 1382 raft_consensus.cc:3037] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:50:08.399250 1794 raft_consensus.cc:3037] T a0003934666140618e1db53138901993 P 3c2f96c5d7a74c168d0489da8797f3e1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:50:08.400579 1794 raft_consensus.cc:2454] T a0003934666140618e1db53138901993 P 3c2f96c5d7a74c168d0489da8797f3e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ed840bbf34b4186b6778095329ee077 in term 1.
I20260810 01:50:08.400594 1382 raft_consensus.cc:2454] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ed840bbf34b4186b6778095329ee077 in term 1.
I20260810 01:50:08.400960 1496 leader_election.cc:302] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c2f96c5d7a74c168d0489da8797f3e1, 5ed840bbf34b4186b6778095329ee077; no voters:
I20260810 01:50:08.401162 1893 raft_consensus.cc:2781] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:50:08.401468 1893 raft_consensus.cc:686] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077 [term 1 LEADER]: Becoming Leader. State: Replica: 5ed840bbf34b4186b6778095329ee077, State: Running, Role: LEADER
I20260810 01:50:08.401934 1893 consensus_queue.cc:237] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } }
I20260810 01:50:08.406026 1166 catalog_manager.cc:5495] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ed840bbf34b4186b6778095329ee077 (127.9.77.130). New cstate: current_term: 1 leader_uuid: "5ed840bbf34b4186b6778095329ee077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } health_report { overall_health: UNKNOWN } } }
I20260810 01:50:08.457902 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:08.453754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50052 (local address 127.9.77.130:38569)
0810 01:50:08.453923 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 01:50:08.453927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:50:08.453944 (+ 17us) server_negotiation.cc:407] Connection header received
0810 01:50:08.454050 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:50:08.454052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:50:08.454117 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:50:08.454224 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:50:08.454611 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:08.455210 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:08.456184 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:08.456438 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:08.456529 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:08.456890 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:50:08.457470 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:50:08.457572 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 01:50:08.457671 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 01:50:08.457735 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 01:50:08.465359 1382 raft_consensus.cc:1261] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e [term 1 FOLLOWER]: Refusing update from remote peer 5ed840bbf34b4186b6778095329ee077: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:50:08.465358 1794 raft_consensus.cc:1261] T a0003934666140618e1db53138901993 P 3c2f96c5d7a74c168d0489da8797f3e1 [term 1 FOLLOWER]: Refusing update from remote peer 5ed840bbf34b4186b6778095329ee077: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:50:08.466236 1900 consensus_queue.cc:1035] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:50:08.466463 1893 consensus_queue.cc:1035] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:50:08.476161 1908 mvcc.cc:205] Tried to move back new op lower bound from 7316793788264800256 to 7316793788017483776. Current Snapshot: MvccSnapshot[applied={T|T < 7316793788264800256}]
I20260810 01:50:08.482436 1907 mvcc.cc:205] Tried to move back new op lower bound from 7316793788264800256 to 7316793788017483776. Current Snapshot: MvccSnapshot[applied={T|T < 7316793788264800256}]
I20260810 01:50:08.485536 1166 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:50:08.485761 1166 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:50:08.489184 1166 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf: Sending DeleteTablet for 3 replicas of tablet a0003934666140618e1db53138901993
I20260810 01:50:08.490525 1761 tablet_service.cc:1514] Processing DeleteTablet for tablet a0003934666140618e1db53138901993 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:50:08 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:37856
I20260810 01:50:08.491606 1568 tablet_service.cc:1514] Processing DeleteTablet for tablet a0003934666140618e1db53138901993 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:50:08 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:50040
I20260810 01:50:08.491869 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 1254
I20260810 01:50:08.501947 1921 tablet_replica.cc:330] T a0003934666140618e1db53138901993 P 3c2f96c5d7a74c168d0489da8797f3e1: stopping tablet replica
I20260810 01:50:08.502527 1921 raft_consensus.cc:2229] T a0003934666140618e1db53138901993 P 3c2f96c5d7a74c168d0489da8797f3e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:50:08.503034 1921 raft_consensus.cc:2258] T a0003934666140618e1db53138901993 P 3c2f96c5d7a74c168d0489da8797f3e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:50:08.504300 1922 tablet_replica.cc:330] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077: stopping tablet replica
I20260810 01:50:08.504698 1922 raft_consensus.cc:2229] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077 [term 1 LEADER]: Raft consensus shutting down.
I20260810 01:50:08.505211 1922 raft_consensus.cc:2258] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:50:08.505926 1921 ts_tablet_manager.cc:1905] T a0003934666140618e1db53138901993 P 3c2f96c5d7a74c168d0489da8797f3e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:50:08.506985 1922 ts_tablet_manager.cc:1905] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:50:08.509495 1921 ts_tablet_manager.cc:1918] T a0003934666140618e1db53138901993 P 3c2f96c5d7a74c168d0489da8797f3e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:50:08.509692 1921 log.cc:1196] T a0003934666140618e1db53138901993 P 3c2f96c5d7a74c168d0489da8797f3e1: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/wal/wals/a0003934666140618e1db53138901993
I20260810 01:50:08.510106 1921 ts_tablet_manager.cc:1939] T a0003934666140618e1db53138901993 P 3c2f96c5d7a74c168d0489da8797f3e1: Deleting consensus metadata
I20260810 01:50:08.510794 1922 ts_tablet_manager.cc:1918] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:50:08.510902 1922 log.cc:1196] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/wal/wals/a0003934666140618e1db53138901993
I20260810 01:50:08.511268 1922 ts_tablet_manager.cc:1939] T a0003934666140618e1db53138901993 P 5ed840bbf34b4186b6778095329ee077: Deleting consensus metadata
I20260810 01:50:08.512239 1151 catalog_manager.cc:4841] TS 3c2f96c5d7a74c168d0489da8797f3e1 (127.9.77.131:46805): tablet a0003934666140618e1db53138901993 (table test-table [id=77e764c0b8054f84a5cf35e9a98ca0fb]) successfully deleted
I20260810 01:50:08.512588 1150 catalog_manager.cc:4841] TS 5ed840bbf34b4186b6778095329ee077 (127.9.77.130:38569): tablet a0003934666140618e1db53138901993 (table test-table [id=77e764c0b8054f84a5cf35e9a98ca0fb]) successfully deleted
W20260810 01:50:08.513428 1151 connection.cc:535] client connection to 127.9.77.129:32833 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 01:50:08.513548 1151 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 01:50:08.513607 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 1461
I20260810 01:50:08.525885 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:08.525457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:32833 (local address 127.0.0.1:55194)
0810 01:50:08.525636 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:08.525733 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.129:32833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260810 01:50:08.526298 1151 catalog_manager.cc:4584] TS 332d12f203e04e69aadf05e9c8ad000e (127.9.77.129:32833): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a0003934666140618e1db53138901993: Network error: Client connection negotiation failed: client connection to 127.9.77.129:32833: connect: Connection refused (error 111)
I20260810 01:50:08.532512 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 1669
I20260810 01:50:08.551391 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 1117
I20260810 01:50:28.573217 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:37329
--webserver_interface=127.9.77.190
--webserver_port=41769
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:37329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:50:28.710304 1926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:50:28.710672 1926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:50:28.710716 1926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:50:28.715005 1926 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 01:50:28.715090 1926 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 01:50:28.715106 1926 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 01:50:28.715121 1926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:50:28.715139 1926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:50:28.715157 1926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:50:28.715211 1926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:50:28.715248 1926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:50:28.721082 1926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:37329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:37329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=41769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:50:28.722626 1926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:50:28.724066 1926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:50:28.742117 1937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:50:28.742100 1935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:50:28.742090 1934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:50:28.742475 1926 server_base.cc:1034] running on GCE node
I20260810 01:50:28.743625 1926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:50:28.745299 1926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:50:28.746531 1926 hybrid_clock.cc:648] HybridClock initialized: now 1786326628746517 us; error 35 us; skew 500 ppm
Aug 10 01:50:28 dist-test-slave-kvfs krb5kdc[1098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326628, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:50:28.755777 1926 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:50:28.757136 1926 webserver.cc:460] Webserver started at http://127.9.77.190:41769/ using document root <none> and password file <none>
I20260810 01:50:28.757936 1926 fs_manager.cc:359] Metadata directory not provided
I20260810 01:50:28.758068 1926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:50:28.763163 1926 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 01:50:28.766125 1946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:50:28.767886 1926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 01:50:28.768133 1926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf"
format_stamp: "Formatted at 2026-08-10 01:50:07 on dist-test-slave-kvfs"
server_key: "9ae0e0b60e96a77c64c6c6a2d8649746"
server_key_iv: "7bd16d3bc162ade345c5df42b9f04ac8"
server_key_version: "encryptionkey@0"
I20260810 01:50:28.768918 1926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:50:28.781920 1926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:50:28.785671 1926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:50:28.785943 1926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:50:28.795634 1926 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:37329
I20260810 01:50:28.796012 2023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:37329 every 8 connection(s)
I20260810 01:50:28.797484 1926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:50:28.803623 2024 sys_catalog.cc:263] Verifying existing consensus state
I20260810 01:50:28.804730 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 1926
I20260810 01:50:28.805285 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:32833
--local_ip_for_outbound_sockets=127.9.77.129
--tserver_master_addrs=127.9.77.190:37329
--webserver_port=36209
--webserver_interface=127.9.77.129
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 01:50:28.806131 2024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf: Bootstrap starting.
I20260810 01:50:28.835162 2024 log.cc:824] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf: Log is configured to *not* fsync() on all Append() calls
I20260810 01:50:28.848582 2024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 01:50:28.849164 2024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf: Bootstrap complete.
I20260810 01:50:28.855619 2024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37329 } }
I20260810 01:50:28.856261 2024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f046c3d7cc5420f9018b43ef4c2c6bf, State: Initialized, Role: FOLLOWER
I20260810 01:50:28.856956 2024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37329 } }
I20260810 01:50:28.857142 2024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:50:28.857215 2024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:50:28.857333 2024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [term 1 FOLLOWER]: Advancing to term 2
I20260810 01:50:28.860677 2024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37329 } }
I20260810 01:50:28.861166 2024 leader_election.cc:302] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f046c3d7cc5420f9018b43ef4c2c6bf; no voters:
I20260810 01:50:28.861620 2024 leader_election.cc:288] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 01:50:28.861836 2031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [term 2 FOLLOWER]: Leader election won for term 2
I20260810 01:50:28.862159 2031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [term 2 LEADER]: Becoming Leader. State: Replica: 2f046c3d7cc5420f9018b43ef4c2c6bf, State: Running, Role: LEADER
I20260810 01:50:28.862555 2031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37329 } }
I20260810 01:50:28.863256 2024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:50:28.864398 2033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37329 } } }
I20260810 01:50:28.864528 2033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [sys.catalog]: This master's current role is: LEADER
I20260810 01:50:28.864368 2035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f046c3d7cc5420f9018b43ef4c2c6bf. Latest consensus state: current_term: 2 leader_uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37329 } } }
I20260810 01:50:28.864678 2035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [sys.catalog]: This master's current role is: LEADER
I20260810 01:50:28.864984 2046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:50:28.870656 2046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:50:28.871392 2046 catalog_manager.cc:1259] Loaded cluster ID: e87b153867594db6ae349e9b0ad636ad
I20260810 01:50:28.871467 2046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:50:28.874688 2046 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:50:28.875555 2046 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf: Loaded TSK: 0
I20260810 01:50:28.891119 2046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf: Generated new TSK 1
I20260810 01:50:28.891472 2046 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 01:50:28.966565 2028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:50:28.966924 2028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:50:28.967010 2028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:50:28.971359 2028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:50:28.971475 2028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:50:28.971580 2028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:50:28.977699 2028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:32833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=36209
--tserver_master_addrs=127.9.77.190:37329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:50:28.979100 2028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:50:28.980470 2028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:50:28.993166 2069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:50:28.993158 2071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:50:28.993551 2067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:50:28.994087 2028 server_base.cc:1034] running on GCE node
I20260810 01:50:28.994612 2028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:50:28.995461 2028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:50:28.996692 2028 hybrid_clock.cc:648] HybridClock initialized: now 1786326628996657 us; error 53 us; skew 500 ppm
Aug 10 01:50:28 dist-test-slave-kvfs krb5kdc[1098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326628, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:50:29.000389 2028 init.cc:377] Logged in from keytab as kudu/127.9.77.129@KRBTEST.COM (short username kudu)
I20260810 01:50:29.001562 2028 webserver.cc:460] Webserver started at http://127.9.77.129:36209/ using document root <none> and password file <none>
I20260810 01:50:29.002296 2028 fs_manager.cc:359] Metadata directory not provided
I20260810 01:50:29.002506 2028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:50:29.007414 2028 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 01:50:29.010177 2081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:50:29.011850 2028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 01:50:29.012010 2028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "332d12f203e04e69aadf05e9c8ad000e"
format_stamp: "Formatted at 2026-08-10 01:50:07 on dist-test-slave-kvfs"
server_key: "2a7d1f09c957083b42f2ee305e3a12c0"
server_key_iv: "6472c5cb9e4fc1b72a69ec99b3129422"
server_key_version: "encryptionkey@0"
I20260810 01:50:29.012784 2028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:50:29.034232 2028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:50:29.038645 2028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:50:29.038914 2028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:50:29.039749 2028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:50:29.041177 2094 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 01:50:29.046479 2028 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 01:50:29.046576 2028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260810 01:50:29.046676 2028 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 01:50:29.048892 2028 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 01:50:29.048965 2028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260810 01:50:29.049327 2094 tablet_bootstrap.cc:492] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e: Bootstrap starting.
I20260810 01:50:29.061728 2028 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:32833
I20260810 01:50:29.061769 2257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:32833 every 8 connection(s)
I20260810 01:50:29.062934 2028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:50:29.068111 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 2028
I20260810 01:50:29.068665 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:38569
--local_ip_for_outbound_sockets=127.9.77.130
--tserver_master_addrs=127.9.77.190:37329
--webserver_port=33415
--webserver_interface=127.9.77.130
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 01:50:29.073675 2094 log.cc:824] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e: Log is configured to *not* fsync() on all Append() calls
Aug 10 01:50:29 dist-test-slave-kvfs krb5kdc[1098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326628, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:50:29.084517 2094 tablet_bootstrap.cc:492] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 01:50:29.085022 2094 tablet_bootstrap.cc:492] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e: Bootstrap complete.
I20260810 01:50:29.086577 2094 ts_tablet_manager.cc:1397] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e: Time spent bootstrapping tablet: real 0.038s user 0.027s sys 0.008s
I20260810 01:50:29.086652 2264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:29.065690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.129:59869)
0810 01:50:29.066460 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:29.066521 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 01:50:29.067513 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:50:29.069297 (+ 1784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:50:29.069309 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:50:29.069940 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:50:29.070742 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:29.070782 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.072913 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.072920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:29.074407 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:29.074422 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.075572 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.075576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:29.075792 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:29.076683 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:50:29.076708 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:50:29.079179 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:50:29.082274 (+ 3095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:50:29.082285 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:50:29.082306 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:50:29.082755 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:50:29.083236 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:50:29.083243 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:50:29.083248 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:50:29.083478 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:50:29.084334 (+ 856us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:50:29.084344 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:50:29.084737 (+ 393us) client_negotiation.cc:769] Sending connection context
0810 01:50:29.084996 (+ 259us) client_negotiation.cc:240] Negotiation successful
0810 01:50:29.085470 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":173,"threads_started":1}
I20260810 01:50:29.087072 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:29.065825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:59869 (local address 127.9.77.190:37329)
0810 01:50:29.066341 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0810 01:50:29.066365 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 01:50:29.066678 (+ 313us) server_negotiation.cc:407] Connection header received
0810 01:50:29.068115 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:50:29.068133 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:50:29.068669 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:50:29.069262 (+ 593us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:50:29.070998 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.072686 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.074556 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.075385 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.075618 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:29.079480 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:50:29.079487 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:50:29.079507 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:50:29.079547 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:50:29.082107 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:50:29.082905 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:50:29.082912 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:50:29.082917 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:50:29.083095 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:50:29.083602 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:50:29.083605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:50:29.083607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:50:29.084172 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:50:29.084311 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 01:50:29.085220 (+ 909us) server_negotiation.cc:299] Negotiation successful
0810 01:50:29.085578 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":145,"threads_started":1}
I20260810 01:50:29.090143 2261 heartbeater.cc:347] Connected to a master server at 127.9.77.190:37329
I20260810 01:50:29.089874 2094 raft_consensus.cc:348] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } }
I20260810 01:50:29.090453 2261 heartbeater.cc:464] Registering TS with master...
I20260810 01:50:29.090505 2094 raft_consensus.cc:729] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 332d12f203e04e69aadf05e9c8ad000e, State: Initialized, Role: FOLLOWER
I20260810 01:50:29.091087 2094 consensus_queue.cc:260] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } }
I20260810 01:50:29.091141 2261 heartbeater.cc:511] Master 127.9.77.190:37329 requested a full tablet report, sending...
I20260810 01:50:29.092281 2094 ts_tablet_manager.cc:1428] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260810 01:50:29.094904 1969 ts_manager.cc:194] Registered new tserver with Master: 332d12f203e04e69aadf05e9c8ad000e (127.9.77.129:32833)
I20260810 01:50:29.100368 1969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.129@KRBTEST.COM'} at 127.9.77.129:59869
I20260810 01:50:29.103770 2261 heartbeater.cc:503] Master 127.9.77.190:37329 was elected leader, sending a full tablet report...
Aug 10 01:50:29 dist-test-slave-kvfs krb5kdc[1098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326628, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for kudu/127.9.77.129@KRBTEST.COM
I20260810 01:50:29.111544 2270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:29.099307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38590 (local address 127.9.77.129:32833)
0810 01:50:29.099713 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0810 01:50:29.099727 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 01:50:29.100186 (+ 459us) server_negotiation.cc:407] Connection header received
0810 01:50:29.100530 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:50:29.100538 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:50:29.100767 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:50:29.100894 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:50:29.101943 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.102717 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.103646 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.104212 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.104343 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:29.107420 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:50:29.107430 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:50:29.107443 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:50:29.107479 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:50:29.109431 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:50:29.110022 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:50:29.110027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:50:29.110028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:50:29.110114 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:50:29.110429 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:50:29.110432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:50:29.110433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:50:29.110714 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:50:29.110839 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 01:50:29.111228 (+ 389us) server_negotiation.cc:299] Negotiation successful
0810 01:50:29.111391 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":185,"threads_started":1}
I20260810 01:50:29.111577 2272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:29.099300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:32833 (local address 127.0.0.1:38590)
0810 01:50:29.100011 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:29.100088 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0810 01:50:29.100316 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:50:29.101122 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:50:29.101130 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:50:29.101685 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:50:29.101818 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:29.101826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.102852 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.102857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:29.103515 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:29.103524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.104364 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.104369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:29.104491 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:29.105237 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:50:29.105262 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:50:29.107244 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:50:29.109553 (+ 2309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:50:29.109565 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:50:29.109571 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:50:29.109897 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:50:29.110203 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:50:29.110207 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:50:29.110209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:50:29.110345 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:50:29.110843 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:50:29.110849 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:50:29.110993 (+ 144us) client_negotiation.cc:769] Sending connection context
0810 01:50:29.111151 (+ 158us) client_negotiation.cc:240] Negotiation successful
0810 01:50:29.111392 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":183,"threads_started":1}
I20260810 01:50:29.112958 2159 tablet_service.cc:1514] Processing DeleteTablet for tablet a0003934666140618e1db53138901993 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:50:08 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:38590
I20260810 01:50:29.114058 2273 tablet_replica.cc:330] stopping tablet replica
I20260810 01:50:29.114480 2273 raft_consensus.cc:2229] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:50:29.114804 2273 raft_consensus.cc:2258] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:50:29.116292 2273 ts_tablet_manager.cc:1905] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:50:29.121893 2273 ts_tablet_manager.cc:1918] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:50:29.122009 2273 log.cc:1196] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/wal/wals/a0003934666140618e1db53138901993
I20260810 01:50:29.122329 2273 ts_tablet_manager.cc:1939] T a0003934666140618e1db53138901993 P 332d12f203e04e69aadf05e9c8ad000e: Deleting consensus metadata
I20260810 01:50:29.124815 1951 catalog_manager.cc:4841] TS 332d12f203e04e69aadf05e9c8ad000e (127.9.77.129:32833): tablet a0003934666140618e1db53138901993 (table test-table [id=77e764c0b8054f84a5cf35e9a98ca0fb]) successfully deleted
W20260810 01:50:29.221321 2266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:50:29.221662 2266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:50:29.221710 2266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:50:29.225801 2266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:50:29.225864 2266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:50:29.226063 2266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:50:29.232759 2266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:38569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=33415
--tserver_master_addrs=127.9.77.190:37329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:50:29.234215 2266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:50:29.235512 2266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:50:29.249563 2278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:50:29.251839 2279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:50:29.252410 2283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:50:29.252756 2266 server_base.cc:1034] running on GCE node
I20260810 01:50:29.253419 2266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:50:29.254596 2266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:50:29.255761 2266 hybrid_clock.cc:648] HybridClock initialized: now 1786326629255677 us; error 103 us; skew 500 ppm
Aug 10 01:50:29 dist-test-slave-kvfs krb5kdc[1098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326629, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:50:29.259194 2266 init.cc:377] Logged in from keytab as kudu/127.9.77.130@KRBTEST.COM (short username kudu)
I20260810 01:50:29.260432 2266 webserver.cc:460] Webserver started at http://127.9.77.130:33415/ using document root <none> and password file <none>
I20260810 01:50:29.261145 2266 fs_manager.cc:359] Metadata directory not provided
I20260810 01:50:29.261263 2266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:50:29.266315 2266 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260810 01:50:29.268882 2293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:50:29.270392 2266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 01:50:29.270531 2266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "5ed840bbf34b4186b6778095329ee077"
format_stamp: "Formatted at 2026-08-10 01:50:07 on dist-test-slave-kvfs"
server_key: "beecc5c2b6695de93ac8a42f736ba0f8"
server_key_iv: "6e32b801a8b77c04597e7bb0dd1413e0"
server_key_version: "encryptionkey@0"
I20260810 01:50:29.271214 2266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:50:29.289067 2266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:50:29.292963 2266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:50:29.293145 2266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:50:29.293800 2266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:50:29.294761 2266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:50:29.294811 2266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:50:29.294857 2266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:50:29.294881 2266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:50:29.305565 2266 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:38569
I20260810 01:50:29.305634 2467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:38569 every 8 connection(s)
I20260810 01:50:29.306797 2266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:50:29.312424 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 2266
I20260810 01:50:29.312942 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:46805
--local_ip_for_outbound_sockets=127.9.77.131
--tserver_master_addrs=127.9.77.190:37329
--webserver_port=43823
--webserver_interface=127.9.77.131
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 01:50:29 dist-test-slave-kvfs krb5kdc[1098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326629, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:50:29.323432 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:29.309160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:48281 (local address 127.9.77.190:37329)
0810 01:50:29.309337 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 01:50:29.309341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:50:29.310815 (+ 1474us) server_negotiation.cc:407] Connection header received
0810 01:50:29.311839 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:50:29.311842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:50:29.311900 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:50:29.311985 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:50:29.313591 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.314514 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.316104 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.316366 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.316459 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:29.319411 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:50:29.319415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:50:29.319417 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:50:29.319446 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:50:29.321128 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:50:29.321684 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:50:29.321687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:50:29.321689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:50:29.321782 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:50:29.322157 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:50:29.322160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:50:29.322162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:50:29.322382 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:50:29.322493 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 01:50:29.323056 (+ 563us) server_negotiation.cc:299] Negotiation successful
0810 01:50:29.323223 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 01:50:29.324486 2472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:29.309615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.130:48281)
0810 01:50:29.310630 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:29.310678 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 01:50:29.311594 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:50:29.312253 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:50:29.312260 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:50:29.312749 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:50:29.313420 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:29.313437 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.314658 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.314661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:29.315914 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:29.315929 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.316539 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.316544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:29.316804 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:29.317554 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:50:29.317577 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:50:29.319217 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:50:29.321254 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:50:29.321261 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:50:29.321278 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:50:29.321572 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:50:29.321877 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:50:29.321880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:50:29.321882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:50:29.322049 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:50:29.322489 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:50:29.322497 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:50:29.322793 (+ 296us) client_negotiation.cc:769] Sending connection context
0810 01:50:29.323001 (+ 208us) client_negotiation.cc:240] Negotiation successful
0810 01:50:29.323390 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"thread_start_us":611,"threads_started":1}
I20260810 01:50:29.326027 2468 heartbeater.cc:347] Connected to a master server at 127.9.77.190:37329
I20260810 01:50:29.326323 2468 heartbeater.cc:464] Registering TS with master...
I20260810 01:50:29.326962 2468 heartbeater.cc:511] Master 127.9.77.190:37329 requested a full tablet report, sending...
I20260810 01:50:29.328336 1969 ts_manager.cc:194] Registered new tserver with Master: 5ed840bbf34b4186b6778095329ee077 (127.9.77.130:38569)
I20260810 01:50:29.329092 1969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.130@KRBTEST.COM'} at 127.9.77.130:48281
W20260810 01:50:29.473289 2474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:50:29.473686 2474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:50:29.473767 2474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:50:29.478369 2474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:50:29.478469 2474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:50:29.478575 2474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:50:29.484581 2474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:46805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=43823
--tserver_master_addrs=127.9.77.190:37329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:50:29.486069 2474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:50:29.487465 2474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:50:29.499449 2481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:50:29.499449 2480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:50:29.499450 2483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:50:29.499495 2474 server_base.cc:1034] running on GCE node
I20260810 01:50:29.500280 2474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:50:29.501138 2474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:50:29.502311 2474 hybrid_clock.cc:648] HybridClock initialized: now 1786326629502292 us; error 41 us; skew 500 ppm
Aug 10 01:50:29 dist-test-slave-kvfs krb5kdc[1098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326629, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:50:29.505645 2474 init.cc:377] Logged in from keytab as kudu/127.9.77.131@KRBTEST.COM (short username kudu)
I20260810 01:50:29.506618 2474 webserver.cc:460] Webserver started at http://127.9.77.131:43823/ using document root <none> and password file <none>
I20260810 01:50:29.507279 2474 fs_manager.cc:359] Metadata directory not provided
I20260810 01:50:29.507359 2474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:50:29.511955 2474 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:50:29.514500 2493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:50:29.516235 2474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 01:50:29.516366 2474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "3c2f96c5d7a74c168d0489da8797f3e1"
format_stamp: "Formatted at 2026-08-10 01:50:07 on dist-test-slave-kvfs"
server_key: "20b51934f150a65e74b3399f11571ce5"
server_key_iv: "d804fdf07a789dbeb2c48a7e617a75e8"
server_key_version: "encryptionkey@0"
I20260810 01:50:29.517124 2474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:50:29.540577 2474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:50:29.544574 2474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:50:29.544759 2474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:50:29.545466 2474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:50:29.546460 2474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:50:29.546511 2474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:50:29.546559 2474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:50:29.546581 2474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:50:29.557327 2474 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:46805
I20260810 01:50:29.557349 2660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:46805 every 8 connection(s)
I20260810 01:50:29.558709 2474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:50:29.565073 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 2474
Aug 10 01:50:29 dist-test-slave-kvfs krb5kdc[1098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326629, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:50:29.575919 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:29.565974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32856 (local address 127.9.77.190:37329)
0810 01:50:29.566237 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0810 01:50:29.566244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:50:29.566435 (+ 191us) server_negotiation.cc:407] Connection header received
0810 01:50:29.566617 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:50:29.566619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:50:29.566698 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:50:29.566802 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:50:29.567702 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.568411 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.569644 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.569972 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.570101 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:29.571685 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:50:29.571689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:50:29.571692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:50:29.571727 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:50:29.573804 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:50:29.574272 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:50:29.574274 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:50:29.574276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:50:29.574393 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:50:29.574762 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:50:29.574768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:50:29.574770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:50:29.575083 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:50:29.575459 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0810 01:50:29.575545 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 01:50:29.575679 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":101,"threads_started":1}
I20260810 01:50:29.576305 2263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:29.560913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:35221 (local address 127.9.77.190:37329)
0810 01:50:29.561079 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 01:50:29.561083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:50:29.562027 (+ 944us) server_negotiation.cc:407] Connection header received
0810 01:50:29.563700 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:50:29.563703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:50:29.563775 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:50:29.563853 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:50:29.565672 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.566568 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.568061 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.568341 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.568436 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:29.571733 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:50:29.571736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:50:29.571738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:50:29.571784 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:50:29.573601 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:50:29.574238 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:50:29.574241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:50:29.574243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:50:29.574378 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:50:29.574790 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:50:29.574794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:50:29.574796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:50:29.575082 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:50:29.575195 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 01:50:29.576010 (+ 815us) server_negotiation.cc:299] Negotiation successful
0810 01:50:29.576162 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 01:50:29.577406 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:29.561318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.131:35221)
0810 01:50:29.561832 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:29.561901 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 01:50:29.563344 (+ 1443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:50:29.564123 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:50:29.564131 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:50:29.564652 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:50:29.565451 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:29.565474 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.566712 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.566715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:29.567887 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:29.567897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.568509 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.568516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:29.568689 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:29.569415 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:50:29.569432 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:50:29.571526 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:50:29.573732 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:50:29.573740 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:50:29.573761 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:50:29.574108 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:50:29.574476 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:50:29.574480 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:50:29.574482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:50:29.574677 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:50:29.575265 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:50:29.575288 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:50:29.575669 (+ 381us) client_negotiation.cc:769] Sending connection context
0810 01:50:29.575966 (+ 297us) client_negotiation.cc:240] Negotiation successful
0810 01:50:29.576389 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":134,"threads_started":1}
I20260810 01:50:29.578603 2662 heartbeater.cc:347] Connected to a master server at 127.9.77.190:37329
I20260810 01:50:29.578872 2662 heartbeater.cc:464] Registering TS with master...
I20260810 01:50:29.579463 2662 heartbeater.cc:511] Master 127.9.77.190:37329 requested a full tablet report, sending...
I20260810 01:50:29.580681 1969 ts_manager.cc:194] Registered new tserver with Master: 3c2f96c5d7a74c168d0489da8797f3e1 (127.9.77.131:46805)
I20260810 01:50:29.581350 1969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.131@KRBTEST.COM'} at 127.9.77.131:35221
I20260810 01:50:29.590399 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 01:50:29.595391 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:29.591219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32860 (local address 127.9.77.190:37329)
0810 01:50:29.591409 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 01:50:29.591413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:50:29.591658 (+ 245us) server_negotiation.cc:407] Connection header received
0810 01:50:29.591831 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:50:29.591833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:50:29.591983 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:50:29.592168 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:50:29.592430 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.593133 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.594084 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.594343 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.594432 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:29.594755 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:50:29.594961 (+ 206us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 01:50:29.595237 (+ 276us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32860: token expired
Metrics: {"server-negotiator.queue_time_us":84}
W20260810 01:50:29.595474 2667 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32860: token expired
W20260810 01:50:29.595914 1868 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.77.190:37329)
I20260810 01:50:29.604550 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:29.596393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32864 (local address 127.9.77.190:37329)
0810 01:50:29.596519 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 01:50:29.596527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 01:50:29.596541 (+ 14us) server_negotiation.cc:407] Connection header received
0810 01:50:29.596644 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:50:29.596646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:50:29.596687 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:50:29.596751 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:50:29.597455 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.598028 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.598906 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.599182 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.599326 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:29.601036 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:50:29.601039 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:50:29.601042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:50:29.601069 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:50:29.602835 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:50:29.603245 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:50:29.603248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:50:29.603261 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:50:29.603353 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:50:29.603665 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:50:29.603668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:50:29.603669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:50:29.603880 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:50:29.603993 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 01:50:29.604294 (+ 301us) server_negotiation.cc:299] Negotiation successful
0810 01:50:29.604404 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 01:50:29.607669 1969 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 01:50:29.611742 1969 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:50:29.621606 2159 tablet_service.cc:1467] Processing CreateTablet for tablet 67b1de4fbb1d4865aceaf99a1ea97ca6 (DEFAULT_TABLE table=test-table [id=cbc45cb6022b47b196bbce33de54084c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:50:29.622517 2159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67b1de4fbb1d4865aceaf99a1ea97ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:50:29.626382 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:29.620104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:38569 (local address 127.0.0.1:56002)
0810 01:50:29.620479 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:29.620501 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 01:50:29.620614 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:50:29.621597 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:50:29.621601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:50:29.621620 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:50:29.621789 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:29.621797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.623350 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.623357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:29.624558 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:29.624568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.625828 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.625832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:29.625977 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:29.626027 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 01:50:29.626116 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 01:50:29.626202 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":24,"thread_start_us":99,"threads_started":1}
I20260810 01:50:29.626435 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:29.620344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56002 (local address 127.9.77.130:38569)
0810 01:50:29.620889 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0810 01:50:29.620901 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 01:50:29.620949 (+ 48us) server_negotiation.cc:407] Connection header received
0810 01:50:29.621060 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:50:29.621068 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:50:29.621387 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:50:29.621975 (+ 588us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:50:29.622218 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.623223 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.624783 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.625700 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.625830 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:29.626025 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0810 01:50:29.626155 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 01:50:29.626261 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":159,"threads_started":1}
I20260810 01:50:29.626806 2680 tablet_bootstrap.cc:492] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e: Bootstrap starting.
I20260810 01:50:29.627946 2680 tablet_bootstrap.cc:654] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e: Neither blocks nor log segments found. Creating new log.
I20260810 01:50:29.628914 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:29.620389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:46805 (local address 127.0.0.1:43790)
0810 01:50:29.620638 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:29.620653 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:50:29.623465 (+ 2812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:50:29.624101 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:50:29.624104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:50:29.624123 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:50:29.624233 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:29.624239 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.625932 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.625936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:29.627344 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:29.627355 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.628583 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.628586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:29.628668 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:29.628690 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 01:50:29.628740 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 01:50:29.628803 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":5,"thread_start_us":114,"threads_started":1}
I20260810 01:50:29.629297 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:29.621094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43790 (local address 127.9.77.131:46805)
0810 01:50:29.621482 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0810 01:50:29.621496 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 01:50:29.621526 (+ 30us) server_negotiation.cc:407] Connection header received
0810 01:50:29.623651 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:50:29.623658 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:50:29.623899 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:50:29.624120 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:50:29.624342 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.625780 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.627490 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.628444 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.628560 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:29.628717 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 01:50:29.628785 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 01:50:29.628884 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":133,"threads_started":1}
I20260810 01:50:29.630683 2368 tablet_service.cc:1467] Processing CreateTablet for tablet 67b1de4fbb1d4865aceaf99a1ea97ca6 (DEFAULT_TABLE table=test-table [id=cbc45cb6022b47b196bbce33de54084c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:50:29.632145 2368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67b1de4fbb1d4865aceaf99a1ea97ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:50:29.632393 2566 tablet_service.cc:1467] Processing CreateTablet for tablet 67b1de4fbb1d4865aceaf99a1ea97ca6 (DEFAULT_TABLE table=test-table [id=cbc45cb6022b47b196bbce33de54084c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:50:29.633234 2680 tablet_bootstrap.cc:492] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e: No bootstrap required, opened a new log
I20260810 01:50:29.633381 2680 ts_tablet_manager.cc:1397] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260810 01:50:29.633541 2566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67b1de4fbb1d4865aceaf99a1ea97ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:50:29.633911 2680 raft_consensus.cc:348] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } }
I20260810 01:50:29.634054 2680 raft_consensus.cc:374] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:50:29.634105 2680 raft_consensus.cc:729] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 332d12f203e04e69aadf05e9c8ad000e, State: Initialized, Role: FOLLOWER
I20260810 01:50:29.634281 2680 consensus_queue.cc:260] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } }
I20260810 01:50:29.634688 2680 ts_tablet_manager.cc:1428] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 01:50:29.642174 2683 tablet_bootstrap.cc:492] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 3c2f96c5d7a74c168d0489da8797f3e1: Bootstrap starting.
I20260810 01:50:29.644464 2686 tablet_bootstrap.cc:492] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077: Bootstrap starting.
I20260810 01:50:29.645051 2683 tablet_bootstrap.cc:654] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 3c2f96c5d7a74c168d0489da8797f3e1: Neither blocks nor log segments found. Creating new log.
I20260810 01:50:29.646797 2683 log.cc:824] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 3c2f96c5d7a74c168d0489da8797f3e1: Log is configured to *not* fsync() on all Append() calls
I20260810 01:50:29.647910 2686 tablet_bootstrap.cc:654] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077: Neither blocks nor log segments found. Creating new log.
I20260810 01:50:29.648943 2686 log.cc:824] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077: Log is configured to *not* fsync() on all Append() calls
I20260810 01:50:29.649698 2683 tablet_bootstrap.cc:492] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 3c2f96c5d7a74c168d0489da8797f3e1: No bootstrap required, opened a new log
I20260810 01:50:29.649978 2683 ts_tablet_manager.cc:1397] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 3c2f96c5d7a74c168d0489da8797f3e1: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 01:50:29.651381 2686 tablet_bootstrap.cc:492] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077: No bootstrap required, opened a new log
I20260810 01:50:29.651614 2686 ts_tablet_manager.cc:1397] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 01:50:29.654949 2683 raft_consensus.cc:348] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 3c2f96c5d7a74c168d0489da8797f3e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } }
I20260810 01:50:29.655346 2683 raft_consensus.cc:374] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 3c2f96c5d7a74c168d0489da8797f3e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:50:29.655421 2683 raft_consensus.cc:729] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 3c2f96c5d7a74c168d0489da8797f3e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c2f96c5d7a74c168d0489da8797f3e1, State: Initialized, Role: FOLLOWER
I20260810 01:50:29.656087 2683 consensus_queue.cc:260] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 3c2f96c5d7a74c168d0489da8797f3e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } }
I20260810 01:50:29.656266 2686 raft_consensus.cc:348] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } }
I20260810 01:50:29.656626 2686 raft_consensus.cc:374] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:50:29.656706 2686 raft_consensus.cc:729] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ed840bbf34b4186b6778095329ee077, State: Initialized, Role: FOLLOWER
I20260810 01:50:29.657083 2662 heartbeater.cc:503] Master 127.9.77.190:37329 was elected leader, sending a full tablet report...
I20260810 01:50:29.657269 2686 consensus_queue.cc:260] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } }
I20260810 01:50:29.657565 2683 ts_tablet_manager.cc:1428] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 3c2f96c5d7a74c168d0489da8797f3e1: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260810 01:50:29.658151 2468 heartbeater.cc:503] Master 127.9.77.190:37329 was elected leader, sending a full tablet report...
I20260810 01:50:29.658427 2686 ts_tablet_manager.cc:1428] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 01:50:29.687889 2691 raft_consensus.cc:482] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:50:29.688256 2691 raft_consensus.cc:504] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } }
I20260810 01:50:29.689783 2691 leader_election.cc:288] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c2f96c5d7a74c168d0489da8797f3e1 (127.9.77.131:46805), 332d12f203e04e69aadf05e9c8ad000e (127.9.77.129:32833)
I20260810 01:50:29.694535 2472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:29.690153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:46805 (local address 127.9.77.130:59497)
0810 01:50:29.690314 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:29.690349 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 01:50:29.690476 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:50:29.690757 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:50:29.690760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:50:29.690783 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:50:29.690970 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:29.690980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.692073 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.692076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:29.693181 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:29.693190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.694071 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.694075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:29.694185 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:29.694214 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 01:50:29.694294 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 01:50:29.694375 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":12}
I20260810 01:50:29.694518 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:29.690286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:59497 (local address 127.9.77.131:46805)
0810 01:50:29.690469 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 01:50:29.690475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:50:29.690492 (+ 17us) server_negotiation.cc:407] Connection header received
0810 01:50:29.690588 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:50:29.690596 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:50:29.690647 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:50:29.690739 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:50:29.691133 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.691928 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.693309 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.693952 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.694054 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:29.694107 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 01:50:29.694321 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 01:50:29.694386 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 01:50:29.695361 2591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b1de4fbb1d4865aceaf99a1ea97ca6" candidate_uuid: "5ed840bbf34b4186b6778095329ee077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" is_pre_election: true
I20260810 01:50:29.695791 2591 raft_consensus.cc:2454] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 3c2f96c5d7a74c168d0489da8797f3e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ed840bbf34b4186b6778095329ee077 in term 0.
I20260810 01:50:29.696010 2693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:29.690536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:32833 (local address 127.9.77.130:52575)
0810 01:50:29.691077 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:29.691093 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:50:29.691211 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:50:29.691541 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:50:29.691545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:50:29.691565 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:50:29.691695 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:29.691701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.692726 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.692728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:29.693771 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:29.693782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.695605 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.695609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:29.695682 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:29.695702 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 01:50:29.695773 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 01:50:29.695826 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":22,"thread_start_us":190,"threads_started":1}
I20260810 01:50:29.696421 2298 leader_election.cc:302] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c2f96c5d7a74c168d0489da8797f3e1, 5ed840bbf34b4186b6778095329ee077; no voters:
I20260810 01:50:29.696112 2694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:29.690700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:52575 (local address 127.9.77.129:32833)
0810 01:50:29.691063 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0810 01:50:29.691085 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 01:50:29.691197 (+ 112us) server_negotiation.cc:407] Connection header received
0810 01:50:29.691302 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:50:29.691304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:50:29.691372 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:50:29.691495 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:50:29.691810 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.692600 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.693910 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.694624 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.694754 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:29.694973 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0810 01:50:29.695824 (+ 851us) server_negotiation.cc:299] Negotiation successful
0810 01:50:29.695937 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":89,"threads_started":1}
I20260810 01:50:29.696725 2691 raft_consensus.cc:2781] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:50:29.696827 2691 raft_consensus.cc:482] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:50:29.696882 2691 raft_consensus.cc:3037] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:50:29.697258 2190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b1de4fbb1d4865aceaf99a1ea97ca6" candidate_uuid: "5ed840bbf34b4186b6778095329ee077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "332d12f203e04e69aadf05e9c8ad000e" is_pre_election: true
I20260810 01:50:29.697613 2190 raft_consensus.cc:2454] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ed840bbf34b4186b6778095329ee077 in term 0.
I20260810 01:50:29.698457 2691 raft_consensus.cc:504] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } }
I20260810 01:50:29.698884 2691 leader_election.cc:288] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077 [CANDIDATE]: Term 1 election: Requested vote from peers 3c2f96c5d7a74c168d0489da8797f3e1 (127.9.77.131:46805), 332d12f203e04e69aadf05e9c8ad000e (127.9.77.129:32833)
I20260810 01:50:29.699343 2591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b1de4fbb1d4865aceaf99a1ea97ca6" candidate_uuid: "5ed840bbf34b4186b6778095329ee077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c2f96c5d7a74c168d0489da8797f3e1"
I20260810 01:50:29.699362 2190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b1de4fbb1d4865aceaf99a1ea97ca6" candidate_uuid: "5ed840bbf34b4186b6778095329ee077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "332d12f203e04e69aadf05e9c8ad000e"
I20260810 01:50:29.699549 2190 raft_consensus.cc:3037] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:50:29.699515 2591 raft_consensus.cc:3037] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 3c2f96c5d7a74c168d0489da8797f3e1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:50:29.700938 2190 raft_consensus.cc:2454] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ed840bbf34b4186b6778095329ee077 in term 1.
I20260810 01:50:29.700938 2591 raft_consensus.cc:2454] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 3c2f96c5d7a74c168d0489da8797f3e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ed840bbf34b4186b6778095329ee077 in term 1.
I20260810 01:50:29.701244 2298 leader_election.cc:302] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 332d12f203e04e69aadf05e9c8ad000e, 5ed840bbf34b4186b6778095329ee077; no voters:
I20260810 01:50:29.701431 2691 raft_consensus.cc:2781] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:50:29.701699 2691 raft_consensus.cc:686] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077 [term 1 LEADER]: Becoming Leader. State: Replica: 5ed840bbf34b4186b6778095329ee077, State: Running, Role: LEADER
I20260810 01:50:29.702178 2691 consensus_queue.cc:237] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } }
I20260810 01:50:29.706329 1969 catalog_manager.cc:5495] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ed840bbf34b4186b6778095329ee077 (127.9.77.130). New cstate: current_term: 1 leader_uuid: "5ed840bbf34b4186b6778095329ee077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } health_report { overall_health: UNKNOWN } } }
I20260810 01:50:29.733011 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:29.728406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56006 (local address 127.9.77.130:38569)
0810 01:50:29.728586 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 01:50:29.728592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:50:29.728612 (+ 20us) server_negotiation.cc:407] Connection header received
0810 01:50:29.728795 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:50:29.728798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:50:29.728878 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:50:29.729030 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:50:29.729349 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.729980 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.731022 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:29.731293 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:29.731435 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:29.731835 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:50:29.732492 (+ 657us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:50:29.732665 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 01:50:29.732719 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 01:50:29.732800 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 01:50:29.741729 2190 raft_consensus.cc:1261] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [term 1 FOLLOWER]: Refusing update from remote peer 5ed840bbf34b4186b6778095329ee077: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:50:29.741732 2591 raft_consensus.cc:1261] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 3c2f96c5d7a74c168d0489da8797f3e1 [term 1 FOLLOWER]: Refusing update from remote peer 5ed840bbf34b4186b6778095329ee077: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:50:29.742652 2691 consensus_queue.cc:1035] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:50:29.742937 2695 consensus_queue.cc:1035] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:50:29.752082 2698 mvcc.cc:205] Tried to move back new op lower bound from 7316793875410272256 to 7316793875263692800. Current Snapshot: MvccSnapshot[applied={T|T < 7316793875410272256}]
I20260810 01:50:29.763234 1967 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:50:29.763449 1967 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:50:29.766431 1967 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf: Sending DeleteTablet for 3 replicas of tablet 67b1de4fbb1d4865aceaf99a1ea97ca6
I20260810 01:50:29.767421 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 2028
I20260810 01:50:29.767379 2159 tablet_service.cc:1514] Processing DeleteTablet for tablet 67b1de4fbb1d4865aceaf99a1ea97ca6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:50:29 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:38590
I20260810 01:50:29.771164 2368 tablet_service.cc:1514] Processing DeleteTablet for tablet 67b1de4fbb1d4865aceaf99a1ea97ca6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:50:29 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:56002
I20260810 01:50:29.780100 2566 tablet_service.cc:1514] Processing DeleteTablet for tablet 67b1de4fbb1d4865aceaf99a1ea97ca6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:50:29 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:43790
I20260810 01:50:29.780848 2719 tablet_replica.cc:330] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 3c2f96c5d7a74c168d0489da8797f3e1: stopping tablet replica
I20260810 01:50:29.781392 2719 raft_consensus.cc:2229] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 3c2f96c5d7a74c168d0489da8797f3e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:50:29.781976 2719 raft_consensus.cc:2258] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 3c2f96c5d7a74c168d0489da8797f3e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:50:29.782712 2718 tablet_replica.cc:330] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077: stopping tablet replica
I20260810 01:50:29.783109 2718 raft_consensus.cc:2229] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077 [term 1 LEADER]: Raft consensus shutting down.
I20260810 01:50:29.783689 2718 raft_consensus.cc:2258] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:50:29.784541 2719 ts_tablet_manager.cc:1905] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 3c2f96c5d7a74c168d0489da8797f3e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:50:29.785260 2718 ts_tablet_manager.cc:1905] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:50:29.788093 2719 ts_tablet_manager.cc:1918] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 3c2f96c5d7a74c168d0489da8797f3e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:50:29.788295 2719 log.cc:1196] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 3c2f96c5d7a74c168d0489da8797f3e1: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/wal/wals/67b1de4fbb1d4865aceaf99a1ea97ca6
I20260810 01:50:29.788770 2719 ts_tablet_manager.cc:1939] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 3c2f96c5d7a74c168d0489da8797f3e1: Deleting consensus metadata
I20260810 01:50:29.789055 2718 ts_tablet_manager.cc:1918] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:50:29.789170 2718 log.cc:1196] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/wal/wals/67b1de4fbb1d4865aceaf99a1ea97ca6
W20260810 01:50:29.789345 1951 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.77.129:32833 (error 108)
I20260810 01:50:29.789536 2718 ts_tablet_manager.cc:1939] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 5ed840bbf34b4186b6778095329ee077: Deleting consensus metadata
I20260810 01:50:29.790706 1951 catalog_manager.cc:4841] TS 3c2f96c5d7a74c168d0489da8797f3e1 (127.9.77.131:46805): tablet 67b1de4fbb1d4865aceaf99a1ea97ca6 (table test-table [id=cbc45cb6022b47b196bbce33de54084c]) successfully deleted
I20260810 01:50:29.790774 1950 catalog_manager.cc:4841] TS 5ed840bbf34b4186b6778095329ee077 (127.9.77.130:38569): tablet 67b1de4fbb1d4865aceaf99a1ea97ca6 (table test-table [id=cbc45cb6022b47b196bbce33de54084c]) successfully deleted
I20260810 01:50:29.791373 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 2266
I20260810 01:50:29.791700 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:29.791297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:32833 (local address 127.0.0.1:38606)
0810 01:50:29.791482 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:29.791576 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.129:32833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260810 01:50:29.806753 1951 catalog_manager.cc:4584] TS 332d12f203e04e69aadf05e9c8ad000e (127.9.77.129:32833): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 67b1de4fbb1d4865aceaf99a1ea97ca6: Network error: Client connection negotiation failed: client connection to 127.9.77.129:32833: connect: Connection refused (error 111)
W20260810 01:50:29.819387 2017 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.9.77.190:37329
I20260810 01:50:29.829311 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 2474
I20260810 01:50:29.842374 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 1926
I20260810 01:50:49.860414 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:37329
--webserver_interface=127.9.77.190
--webserver_port=41769
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:37329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:50:50.003226 2724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:50:50.003703 2724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:50:50.003804 2724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:50:50.008415 2724 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 01:50:50.008520 2724 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 01:50:50.008553 2724 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 01:50:50.008581 2724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:50:50.008621 2724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:50:50.008678 2724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:50:50.008721 2724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:50:50.008755 2724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:50:50.015425 2724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:37329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:37329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=41769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:50:50.017113 2724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:50:50.018487 2724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:50:50.029716 2731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:50:50.029819 2733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:50:50.029896 2724 server_base.cc:1034] running on GCE node
W20260810 01:50:50.029790 2729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:50:50.030723 2724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:50:50.032310 2724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:50:50.033509 2724 hybrid_clock.cc:648] HybridClock initialized: now 1786326650033478 us; error 50 us; skew 500 ppm
Aug 10 01:50:50 dist-test-slave-kvfs krb5kdc[1098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326650, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:50:50.037153 2724 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:50:50.038281 2724 webserver.cc:460] Webserver started at http://127.9.77.190:41769/ using document root <none> and password file <none>
I20260810 01:50:50.039036 2724 fs_manager.cc:359] Metadata directory not provided
I20260810 01:50:50.039148 2724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:50:50.044339 2724 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:50:50.047071 2747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:50:50.048657 2724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:50:50.048840 2724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf"
format_stamp: "Formatted at 2026-08-10 01:50:07 on dist-test-slave-kvfs"
server_key: "9ae0e0b60e96a77c64c6c6a2d8649746"
server_key_iv: "7bd16d3bc162ade345c5df42b9f04ac8"
server_key_version: "encryptionkey@0"
I20260810 01:50:50.049568 2724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:50:50.079200 2724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:50:50.083913 2724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:50:50.084245 2724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:50:50.093568 2724 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:37329
I20260810 01:50:50.093942 2819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:37329 every 8 connection(s)
I20260810 01:50:50.095091 2724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:50:50.101161 2821 sys_catalog.cc:263] Verifying existing consensus state
I20260810 01:50:50.102788 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 2724
I20260810 01:50:50.103389 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:32833
--local_ip_for_outbound_sockets=127.9.77.129
--tserver_master_addrs=127.9.77.190:37329
--webserver_port=36209
--webserver_interface=127.9.77.129
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 01:50:50.103753 2821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf: Bootstrap starting.
I20260810 01:50:50.133670 2821 log.cc:824] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf: Log is configured to *not* fsync() on all Append() calls
I20260810 01:50:50.151072 2821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 01:50:50.151729 2821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf: Bootstrap complete.
I20260810 01:50:50.158715 2821 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37329 } }
I20260810 01:50:50.159328 2821 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f046c3d7cc5420f9018b43ef4c2c6bf, State: Initialized, Role: FOLLOWER
I20260810 01:50:50.160133 2821 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37329 } }
I20260810 01:50:50.160306 2821 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:50:50.160401 2821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:50:50.160517 2821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [term 2 FOLLOWER]: Advancing to term 3
I20260810 01:50:50.163583 2821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37329 } }
I20260810 01:50:50.164070 2821 leader_election.cc:302] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f046c3d7cc5420f9018b43ef4c2c6bf; no voters:
I20260810 01:50:50.164544 2821 leader_election.cc:288] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 01:50:50.164700 2826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [term 3 FOLLOWER]: Leader election won for term 3
I20260810 01:50:50.165041 2826 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [term 3 LEADER]: Becoming Leader. State: Replica: 2f046c3d7cc5420f9018b43ef4c2c6bf, State: Running, Role: LEADER
I20260810 01:50:50.165467 2826 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37329 } }
I20260810 01:50:50.166191 2821 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:50:50.167281 2828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f046c3d7cc5420f9018b43ef4c2c6bf. Latest consensus state: current_term: 3 leader_uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37329 } } }
I20260810 01:50:50.167443 2828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [sys.catalog]: This master's current role is: LEADER
I20260810 01:50:50.167281 2827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f046c3d7cc5420f9018b43ef4c2c6bf" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37329 } } }
I20260810 01:50:50.167620 2827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf [sys.catalog]: This master's current role is: LEADER
I20260810 01:50:50.168216 2837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:50:50.174940 2837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:50:50.175876 2837 catalog_manager.cc:1259] Loaded cluster ID: e87b153867594db6ae349e9b0ad636ad
I20260810 01:50:50.175966 2837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:50:50.178061 2837 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:50:50.178750 2837 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf: Loaded TSK: 0
I20260810 01:50:50.178860 2837 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2f046c3d7cc5420f9018b43ef4c2c6bf: Loaded TSK: 1
F20260810 01:50:50.179639 2837 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc5b63189ad google::LogMessage::Fail() at ??:0
@ 0x7fc5b631b88a google::LogMessage::SendToLog() at ??:0
@ 0x7fc5b63184de google::LogMessage::Flush() at ??:0
@ 0x7fc5b6319a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc5b8fdc18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc5b8fd4f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fc5c0c817bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fc5c0c5b99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fc5c0c5c178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fc5c0c9ddf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fc5c0cbc82b std::function<>::operator()() at ??:0
@ 0x7fc5c0c5bb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fc5c0c5cf3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc5c0c59579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fc5c0c9d640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc5b856a3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc5b856ac83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fc5b856c3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc5b855b512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc5b9b9f6db start_thread at ??:0
@ 0x7fc5b4e1661f clone at ??:0
W20260810 01:50:50.309633 2823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:50:50.310007 2823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:50:50.310056 2823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:50:50.314502 2823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:50:50.314592 2823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:50:50.314718 2823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:50:50.321018 2823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:32833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=36209
--tserver_master_addrs=127.9.77.190:37329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:50:50.322654 2823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:50:50.324189 2823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:50:50.336668 2890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:50:50.336661 2891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:50:50.337108 2823 server_base.cc:1034] running on GCE node
W20260810 01:50:50.337353 2894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:50:50.337927 2823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:50:50.338773 2823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:50:50.339952 2823 hybrid_clock.cc:648] HybridClock initialized: now 1786326650339921 us; error 53 us; skew 500 ppm
Aug 10 01:50:50 dist-test-slave-kvfs krb5kdc[1098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326650, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:50:50.343358 2823 init.cc:377] Logged in from keytab as kudu/127.9.77.129@KRBTEST.COM (short username kudu)
I20260810 01:50:50.344630 2823 webserver.cc:460] Webserver started at http://127.9.77.129:36209/ using document root <none> and password file <none>
I20260810 01:50:50.345360 2823 fs_manager.cc:359] Metadata directory not provided
I20260810 01:50:50.345430 2823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:50:50.350593 2823 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 01:50:50.353852 2902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:50:50.355628 2823 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 01:50:50.355814 2823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "332d12f203e04e69aadf05e9c8ad000e"
format_stamp: "Formatted at 2026-08-10 01:50:07 on dist-test-slave-kvfs"
server_key: "2a7d1f09c957083b42f2ee305e3a12c0"
server_key_iv: "6472c5cb9e4fc1b72a69ec99b3129422"
server_key_version: "encryptionkey@0"
I20260810 01:50:50.356693 2823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:50:50.378973 2823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:50:50.383518 2823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:50:50.383816 2823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:50:50.384722 2823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:50:50.386250 2911 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 01:50:50.393638 2823 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 01:50:50.393813 2823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.000s
I20260810 01:50:50.393936 2823 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 01:50:50.396405 2823 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 01:50:50.396571 2823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260810 01:50:50.396911 2911 tablet_bootstrap.cc:492] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e: Bootstrap starting.
I20260810 01:50:50.410043 2823 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:32833
I20260810 01:50:50.410374 3066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:32833 every 8 connection(s)
I20260810 01:50:50.411381 2823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:50:50.417500 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 2823
I20260810 01:50:50.418125 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:38569
--local_ip_for_outbound_sockets=127.9.77.130
--tserver_master_addrs=127.9.77.190:37329
--webserver_port=33415
--webserver_interface=127.9.77.130
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 01:50:50.422500 2911 log.cc:824] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e: Log is configured to *not* fsync() on all Append() calls
I20260810 01:50:50.435699 2911 tablet_bootstrap.cc:492] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 01:50:50.436299 2911 tablet_bootstrap.cc:492] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e: Bootstrap complete.
I20260810 01:50:50.438189 2911 ts_tablet_manager.cc:1397] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e: Time spent bootstrapping tablet: real 0.042s user 0.027s sys 0.012s
I20260810 01:50:50.442301 2911 raft_consensus.cc:348] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } }
I20260810 01:50:50.442930 2911 raft_consensus.cc:729] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 332d12f203e04e69aadf05e9c8ad000e, State: Initialized, Role: FOLLOWER
I20260810 01:50:50.443609 2911 consensus_queue.cc:260] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } }
I20260810 01:50:50.446393 2911 ts_tablet_manager.cc:1428] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e: Time spent starting tablet: real 0.008s user 0.006s sys 0.003s
W20260810 01:50:50.458782 3069 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:50:50.414280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.129:56869)
0810 01:50:50.415147 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:50.415221 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0810 01:50:50.416566 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:50:50.457470 (+ 40904us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":277,"threads_started":1}
W20260810 01:50:50.459497 3067 heartbeater.cc:644] Failed to heartbeat to 127.9.77.190:37329 (0 consecutive failures): Network error: Failed to ping master at 127.9.77.190:37329: Client connection negotiation failed: client connection to 127.9.77.190:37329: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 01:50:50.460518 3069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:50.460168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.129:36569)
0810 01:50:50.460336 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:50.460434 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 01:50:50.461308 3069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:50.461078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.129:54423)
0810 01:50:50.461188 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:50.461249 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 01:50:50.461529 3067 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 01:50:50.570320 3072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:50:50.570657 3072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:50:50.570706 3072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:50:50.574981 3072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:50:50.575055 3072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:50:50.575155 3072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:50:50.580974 3072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:38569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=33415
--tserver_master_addrs=127.9.77.190:37329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:50:50.582517 3072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:50:50.583885 3072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:50:50.596695 3081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:50:50.596765 3084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:50:50.596781 3080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:50:50.596850 3072 server_base.cc:1034] running on GCE node
I20260810 01:50:50.597527 3072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:50:50.598331 3072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:50:50.599534 3072 hybrid_clock.cc:648] HybridClock initialized: now 1786326650599514 us; error 39 us; skew 500 ppm
Aug 10 01:50:50 dist-test-slave-kvfs krb5kdc[1098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326650, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:50:50.603005 3072 init.cc:377] Logged in from keytab as kudu/127.9.77.130@KRBTEST.COM (short username kudu)
I20260810 01:50:50.604118 3072 webserver.cc:460] Webserver started at http://127.9.77.130:33415/ using document root <none> and password file <none>
I20260810 01:50:50.604803 3072 fs_manager.cc:359] Metadata directory not provided
I20260810 01:50:50.604871 3072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:50:50.610026 3072 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 01:50:50.613155 3094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:50:50.615059 3072 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 01:50:50.615219 3072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "5ed840bbf34b4186b6778095329ee077"
format_stamp: "Formatted at 2026-08-10 01:50:07 on dist-test-slave-kvfs"
server_key: "beecc5c2b6695de93ac8a42f736ba0f8"
server_key_iv: "6e32b801a8b77c04597e7bb0dd1413e0"
server_key_version: "encryptionkey@0"
I20260810 01:50:50.616112 3072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:50:50.651203 3072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:50:50.655740 3072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:50:50.655949 3072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:50:50.656798 3072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:50:50.657886 3072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:50:50.657949 3072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:50:50.658031 3072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:50:50.658071 3072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:50:50.669728 3072 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:38569
I20260810 01:50:50.669752 3263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:38569 every 8 connection(s)
I20260810 01:50:50.671056 3072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:50:50.673942 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 3072
I20260810 01:50:50.674472 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:46805
--local_ip_for_outbound_sockets=127.9.77.131
--tserver_master_addrs=127.9.77.190:37329
--webserver_port=43823
--webserver_interface=127.9.77.131
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 01:50:50.676352 3267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:50.673906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.130:46679)
0810 01:50:50.674584 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:50.674745 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":218,"threads_started":1}
W20260810 01:50:50.677089 3264 heartbeater.cc:644] Failed to heartbeat to 127.9.77.190:37329 (0 consecutive failures): Network error: Failed to ping master at 127.9.77.190:37329: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
I20260810 01:50:50.677850 3267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:50.677566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.130:54087)
0810 01:50:50.677699 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:50.677784 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 01:50:50.678639 3267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:50.678347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.130:54217)
0810 01:50:50.678479 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:50.678557 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 01:50:50.678910 3264 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 01:50:50.815961 3268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:50:50.816457 3268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:50:50.816543 3268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:50:50.821074 3268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:50:50.821168 3268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:50:50.821259 3268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:50:50.827375 3268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:46805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=43823
--tserver_master_addrs=127.9.77.190:37329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:50:50.829046 3268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:50:50.830480 3268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:50:50.842978 3280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:50:50.842983 3276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:50:50.843027 3277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:50:50.843389 3268 server_base.cc:1034] running on GCE node
I20260810 01:50:50.843952 3268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:50:50.844877 3268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:50:50.846092 3268 hybrid_clock.cc:648] HybridClock initialized: now 1786326650846070 us; error 34 us; skew 500 ppm
Aug 10 01:50:50 dist-test-slave-kvfs krb5kdc[1098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326650, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:50:50.849584 3268 init.cc:377] Logged in from keytab as kudu/127.9.77.131@KRBTEST.COM (short username kudu)
I20260810 01:50:50.850711 3268 webserver.cc:460] Webserver started at http://127.9.77.131:43823/ using document root <none> and password file <none>
I20260810 01:50:50.851438 3268 fs_manager.cc:359] Metadata directory not provided
I20260810 01:50:50.851564 3268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:50:50.856712 3268 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:50:50.859447 3290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:50:50.861378 3268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:50:50.861553 3268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "3c2f96c5d7a74c168d0489da8797f3e1"
format_stamp: "Formatted at 2026-08-10 01:50:07 on dist-test-slave-kvfs"
server_key: "20b51934f150a65e74b3399f11571ce5"
server_key_iv: "d804fdf07a789dbeb2c48a7e617a75e8"
server_key_version: "encryptionkey@0"
I20260810 01:50:50.862361 3268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:50:50.874639 3268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:50:50.878314 3268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:50:50.878566 3268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:50:50.879328 3268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:50:50.880429 3268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:50:50.880506 3268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:50:50.880589 3268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:50:50.880645 3268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:50:50.891932 3268 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:46805
I20260810 01:50:50.891961 3457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:46805 every 8 connection(s)
I20260810 01:50:50.893384 3268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:50:50.895473 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 3268
I20260810 01:50:50.898828 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:50.896426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.131:60053)
0810 01:50:50.897137 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:50.897276 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":160,"threads_started":1}
W20260810 01:50:50.899382 3458 heartbeater.cc:644] Failed to heartbeat to 127.9.77.190:37329 (0 consecutive failures): Network error: Failed to ping master at 127.9.77.190:37329: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
I20260810 01:50:50.900182 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:50.899885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.131:57741)
0810 01:50:50.900043 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:50.900105 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 01:50:50.900897 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:50.900665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.131:58251)
0810 01:50:50.900782 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:50.900833 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 01:50:50.901108 3458 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 01:50:51.462837 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:51.462222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.129:60341)
0810 01:50:51.462623 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:51.462690 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":115,"threads_started":1}
I20260810 01:50:51.680181 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:51.679611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.130:49071)
0810 01:50:51.679921 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:51.679986 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":95,"threads_started":1}
I20260810 01:50:51.902226 3464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:51.901716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.131:38181)
0810 01:50:51.902019 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:51.902087 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":95,"threads_started":1}
I20260810 01:50:52.059084 3466 raft_consensus.cc:482] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:50:52.059440 3466 raft_consensus.cc:504] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } }
I20260810 01:50:52.061043 3466 leader_election.cc:288] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ed840bbf34b4186b6778095329ee077 (127.9.77.130:38569), 3c2f96c5d7a74c168d0489da8797f3e1 (127.9.77.131:46805)
Aug 10 01:50:52 dist-test-slave-kvfs krb5kdc[1098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326650, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for kudu/127.9.77.131@KRBTEST.COM
Aug 10 01:50:52 dist-test-slave-kvfs krb5kdc[1098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326650, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for kudu/127.9.77.130@KRBTEST.COM
I20260810 01:50:52.081677 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:52.061674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:58881 (local address 127.9.77.130:38569)
0810 01:50:52.062205 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0810 01:50:52.062242 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0810 01:50:52.062303 (+ 61us) server_negotiation.cc:407] Connection header received
0810 01:50:52.063423 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:50:52.063434 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:50:52.064056 (+ 622us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:50:52.064484 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:50:52.066538 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:52.068366 (+ 1828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:52.070838 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:52.071679 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:52.071887 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:52.076317 (+ 4430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:50:52.076327 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:50:52.076345 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:50:52.076415 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:50:52.078606 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:50:52.079274 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:50:52.079280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:50:52.079285 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:50:52.079430 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:50:52.079801 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:50:52.079806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:50:52.079808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:50:52.080380 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:50:52.080521 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 01:50:52.081112 (+ 591us) server_negotiation.cc:299] Negotiation successful
0810 01:50:52.081440 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":114,"threads_started":1}
I20260810 01:50:52.081676 3468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:52.061410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:38569 (local address 127.9.77.129:58881)
0810 01:50:52.061936 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:52.061955 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 01:50:52.062131 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:50:52.064614 (+ 2483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:50:52.064623 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:50:52.065423 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:50:52.066309 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:52.066315 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:52.068560 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:52.068564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:52.070671 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:52.070686 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:52.071857 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:52.071862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:52.072272 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:52.073193 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:50:52.073243 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:50:52.076088 (+ 2845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:50:52.078737 (+ 2649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:50:52.078741 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:50:52.078743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:50:52.079140 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:50:52.079532 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:50:52.079537 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:50:52.079539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:50:52.079704 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:50:52.080499 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:50:52.080506 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:50:52.080774 (+ 268us) client_negotiation.cc:769] Sending connection context
0810 01:50:52.080988 (+ 214us) client_negotiation.cc:240] Negotiation successful
0810 01:50:52.081440 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":740,"spinlock_wait_cycles":94592,"thread_start_us":76,"threads_started":1}
I20260810 01:50:52.081676 3469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:52.061674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:44471 (local address 127.9.77.131:46805)
0810 01:50:52.062205 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0810 01:50:52.062242 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0810 01:50:52.062303 (+ 61us) server_negotiation.cc:407] Connection header received
0810 01:50:52.063423 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:50:52.063434 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:50:52.064056 (+ 622us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:50:52.064484 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:50:52.066538 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:52.068366 (+ 1828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:52.070838 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:50:52.071679 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:50:52.071887 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:52.076086 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:50:52.076095 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:50:52.076121 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:50:52.076175 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:50:52.078557 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:50:52.079285 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:50:52.079290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:50:52.079294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:50:52.079429 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:50:52.079847 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:50:52.079850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:50:52.079852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:50:52.080372 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:50:52.080518 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 01:50:52.081113 (+ 595us) server_negotiation.cc:299] Negotiation successful
0810 01:50:52.081423 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":92,"threads_started":1}
I20260810 01:50:52.081698 3467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:52.061410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:46805 (local address 127.9.77.129:44471)
0810 01:50:52.061936 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:52.061955 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 01:50:52.062131 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:50:52.064614 (+ 2483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:50:52.064623 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:50:52.065423 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:50:52.066287 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:52.066313 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:52.068560 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:52.068564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:52.070671 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:50:52.070686 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:50:52.071857 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:50:52.071862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:50:52.072228 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:50:52.073193 (+ 965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:50:52.073226 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:50:52.075722 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:50:52.078705 (+ 2983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:50:52.078713 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:50:52.078730 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:50:52.079138 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:50:52.079540 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:50:52.079544 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:50:52.079546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:50:52.079712 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:50:52.080499 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:50:52.080506 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:50:52.080776 (+ 270us) client_negotiation.cc:769] Sending connection context
0810 01:50:52.080987 (+ 211us) client_negotiation.cc:240] Negotiation successful
0810 01:50:52.081529 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":417,"spinlock_wait_cycles":65536,"thread_start_us":91,"threads_started":1}
W20260810 01:50:52.085242 2907 leader_election.cc:341] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c2f96c5d7a74c168d0489da8797f3e1 (127.9.77.131:46805): Not found: Tablet not found: 67b1de4fbb1d4865aceaf99a1ea97ca6
W20260810 01:50:52.085484 2906 leader_election.cc:341] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ed840bbf34b4186b6778095329ee077 (127.9.77.130:38569): Not found: Tablet not found: 67b1de4fbb1d4865aceaf99a1ea97ca6
I20260810 01:50:52.085574 2906 leader_election.cc:302] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 332d12f203e04e69aadf05e9c8ad000e; no voters: 3c2f96c5d7a74c168d0489da8797f3e1, 5ed840bbf34b4186b6778095329ee077
I20260810 01:50:52.085896 3466 raft_consensus.cc:2726] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 01:50:52.464131 3468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:52.463719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.129:59471)
0810 01:50:52.463893 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:52.463982 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 01:50:52.681586 3473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:52.681072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.130:46697)
0810 01:50:52.681377 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:52.681439 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":106,"threads_started":1}
I20260810 01:50:52.903605 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:52.903122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.131:48905)
0810 01:50:52.903420 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:52.903481 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":102,"threads_started":1}
I20260810 01:50:53.465771 3475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:53.465199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.129:50293)
0810 01:50:53.465550 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:53.465626 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":114,"threads_started":1}
I20260810 01:50:53.603617 3476 raft_consensus.cc:482] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:50:53.603833 3476 raft_consensus.cc:504] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } }
I20260810 01:50:53.604452 3476 leader_election.cc:288] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ed840bbf34b4186b6778095329ee077 (127.9.77.130:38569), 3c2f96c5d7a74c168d0489da8797f3e1 (127.9.77.131:46805)
W20260810 01:50:53.605453 2907 leader_election.cc:341] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c2f96c5d7a74c168d0489da8797f3e1 (127.9.77.131:46805): Not found: Tablet not found: 67b1de4fbb1d4865aceaf99a1ea97ca6
W20260810 01:50:53.605587 2906 leader_election.cc:341] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ed840bbf34b4186b6778095329ee077 (127.9.77.130:38569): Not found: Tablet not found: 67b1de4fbb1d4865aceaf99a1ea97ca6
I20260810 01:50:53.605726 2906 leader_election.cc:302] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 332d12f203e04e69aadf05e9c8ad000e; no voters: 3c2f96c5d7a74c168d0489da8797f3e1, 5ed840bbf34b4186b6778095329ee077
I20260810 01:50:53.605885 3476 raft_consensus.cc:2726] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 01:50:53.683039 3477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:53.682458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.130:35829)
0810 01:50:53.682813 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:53.682885 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":124,"threads_started":1}
I20260810 01:50:53.905028 3478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:53.904545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.131:56225)
0810 01:50:53.904838 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:53.904905 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":100,"threads_started":1}
I20260810 01:50:54.467595 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:54.466929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.129:56675)
0810 01:50:54.467295 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:54.467396 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":124,"threads_started":1}
I20260810 01:50:54.684618 3481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:54.684004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.130:49411)
0810 01:50:54.684404 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:54.684484 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":122,"threads_started":1}
I20260810 01:50:54.906538 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:54.905993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.131:49457)
0810 01:50:54.906335 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:54.906408 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":112,"threads_started":1}
I20260810 01:50:55.469316 3485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:55.468727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.129:38227)
0810 01:50:55.469109 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:55.469181 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":102,"threads_started":1}
I20260810 01:50:55.686383 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:55.685761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.130:50473)
0810 01:50:55.686116 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:55.686192 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":127,"threads_started":1}
I20260810 01:50:55.908315 3487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:55.907663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.131:38117)
0810 01:50:55.908091 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:55.908176 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":117,"threads_started":1}
I20260810 01:50:56.470790 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:56.470233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.129:43747)
0810 01:50:56.470583 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:56.470652 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":142,"threads_started":1}
I20260810 01:50:56.688118 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:56.687439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.130:34141)
0810 01:50:56.687809 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:56.687887 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":145,"threads_started":1}
I20260810 01:50:56.837530 3493 raft_consensus.cc:482] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:50:56.837826 3493 raft_consensus.cc:504] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } }
I20260810 01:50:56.838485 3493 leader_election.cc:288] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ed840bbf34b4186b6778095329ee077 (127.9.77.130:38569), 3c2f96c5d7a74c168d0489da8797f3e1 (127.9.77.131:46805)
W20260810 01:50:56.839569 2906 leader_election.cc:341] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ed840bbf34b4186b6778095329ee077 (127.9.77.130:38569): Not found: Tablet not found: 67b1de4fbb1d4865aceaf99a1ea97ca6
W20260810 01:50:56.839749 2907 leader_election.cc:341] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c2f96c5d7a74c168d0489da8797f3e1 (127.9.77.131:46805): Not found: Tablet not found: 67b1de4fbb1d4865aceaf99a1ea97ca6
I20260810 01:50:56.839974 2907 leader_election.cc:302] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 332d12f203e04e69aadf05e9c8ad000e; no voters: 3c2f96c5d7a74c168d0489da8797f3e1, 5ed840bbf34b4186b6778095329ee077
I20260810 01:50:56.840229 3493 raft_consensus.cc:2726] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 01:50:56.909984 3496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:56.909449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.131:54799)
0810 01:50:56.909783 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:56.909843 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":138,"threads_started":1}
I20260810 01:50:57.472594 3498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:57.471930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.129:44277)
0810 01:50:57.472360 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:57.472445 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":185,"threads_started":1}
I20260810 01:50:57.689704 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:57.689066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.130:37511)
0810 01:50:57.689488 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:57.689564 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":104,"threads_started":1}
I20260810 01:50:57.911629 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:57.910991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.131:34481)
0810 01:50:57.911427 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:57.911497 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":130,"threads_started":1}
I20260810 01:50:58.474176 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:58.473608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.129:49525)
0810 01:50:58.473966 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:58.474038 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":103,"threads_started":1}
I20260810 01:50:58.691224 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:58.690681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.130:40169)
0810 01:50:58.691017 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:58.691090 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":108,"threads_started":1}
I20260810 01:50:58.913056 3503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:58.912535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.131:39007)
0810 01:50:58.912850 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:58.912924 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":123,"threads_started":1}
I20260810 01:50:59.475801 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:59.475246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.129:59125)
0810 01:50:59.475574 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:59.475644 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":110,"threads_started":1}
I20260810 01:50:59.692982 3505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:59.692366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.130:47069)
0810 01:50:59.692758 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:59.692841 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":106,"threads_started":1}
I20260810 01:50:59.839511 3506 raft_consensus.cc:482] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:50:59.839730 3506 raft_consensus.cc:504] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } }
I20260810 01:50:59.840397 3506 leader_election.cc:288] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ed840bbf34b4186b6778095329ee077 (127.9.77.130:38569), 3c2f96c5d7a74c168d0489da8797f3e1 (127.9.77.131:46805)
W20260810 01:50:59.841317 2907 leader_election.cc:341] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c2f96c5d7a74c168d0489da8797f3e1 (127.9.77.131:46805): Not found: Tablet not found: 67b1de4fbb1d4865aceaf99a1ea97ca6
W20260810 01:50:59.841471 2906 leader_election.cc:341] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ed840bbf34b4186b6778095329ee077 (127.9.77.130:38569): Not found: Tablet not found: 67b1de4fbb1d4865aceaf99a1ea97ca6
I20260810 01:50:59.841563 2906 leader_election.cc:302] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 332d12f203e04e69aadf05e9c8ad000e; no voters: 3c2f96c5d7a74c168d0489da8797f3e1, 5ed840bbf34b4186b6778095329ee077
I20260810 01:50:59.841738 3506 raft_consensus.cc:2726] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 01:50:59.914507 3507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:50:59.913975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.131:51709)
0810 01:50:59.914305 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0810 01:50:59.914377 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":107,"threads_started":1}
I20260810 01:51:00.477742 3508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:00.477111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.129:46153)
0810 01:51:00.477515 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:00.477601 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":117,"threads_started":1}
I20260810 01:51:00.694734 3509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:00.694105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.130:59693)
0810 01:51:00.694472 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:00.694564 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":108,"threads_started":1}
I20260810 01:51:00.916342 3510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:00.915662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.131:38619)
0810 01:51:00.916106 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:00.916186 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":129,"threads_started":1}
I20260810 01:51:01.479362 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:01.478721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.129:48449)
0810 01:51:01.479145 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:01.479221 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":104,"threads_started":1}
I20260810 01:51:01.696251 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:01.695678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.130:54827)
0810 01:51:01.696000 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:01.696114 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":120,"threads_started":1}
I20260810 01:51:01.917971 3515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:01.917414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.131:37567)
0810 01:51:01.917750 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:01.917820 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":116,"threads_started":1}
I20260810 01:51:02.481158 3516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:02.480473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.129:60179)
0810 01:51:02.480913 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:02.481010 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":127,"threads_started":1}
I20260810 01:51:02.698104 3517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:02.697251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.130:47717)
0810 01:51:02.697899 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:02.697967 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":134,"threads_started":1}
I20260810 01:51:02.919636 3519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:02.919072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.131:46623)
0810 01:51:02.919430 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:02.919501 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":112,"threads_started":1}
I20260810 01:51:03.482836 3521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:03.482315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.129:38989)
0810 01:51:03.482632 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:03.482696 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":105,"threads_started":1}
I20260810 01:51:03.699877 3524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:03.699227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.130:42905)
0810 01:51:03.699666 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:03.699740 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":144,"threads_started":1}
I20260810 01:51:03.921262 3526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:03.920683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.131:50391)
0810 01:51:03.921050 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:03.921122 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":114,"threads_started":1}
I20260810 01:51:04.196561 3527 raft_consensus.cc:482] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:51:04.196774 3527 raft_consensus.cc:504] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ed840bbf34b4186b6778095329ee077" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 38569 } } peers { permanent_uuid: "3c2f96c5d7a74c168d0489da8797f3e1" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 46805 } } peers { permanent_uuid: "332d12f203e04e69aadf05e9c8ad000e" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 32833 } }
I20260810 01:51:04.197327 3527 leader_election.cc:288] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ed840bbf34b4186b6778095329ee077 (127.9.77.130:38569), 3c2f96c5d7a74c168d0489da8797f3e1 (127.9.77.131:46805)
W20260810 01:51:04.198441 2906 leader_election.cc:341] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ed840bbf34b4186b6778095329ee077 (127.9.77.130:38569): Not found: Tablet not found: 67b1de4fbb1d4865aceaf99a1ea97ca6
W20260810 01:51:04.198596 2907 leader_election.cc:341] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c2f96c5d7a74c168d0489da8797f3e1 (127.9.77.131:46805): Not found: Tablet not found: 67b1de4fbb1d4865aceaf99a1ea97ca6
I20260810 01:51:04.198674 2907 leader_election.cc:302] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 332d12f203e04e69aadf05e9c8ad000e; no voters: 3c2f96c5d7a74c168d0489da8797f3e1, 5ed840bbf34b4186b6778095329ee077
I20260810 01:51:04.198829 3527 raft_consensus.cc:2726] T 67b1de4fbb1d4865aceaf99a1ea97ca6 P 332d12f203e04e69aadf05e9c8ad000e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 01:51:04.484429 3528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:04.483781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.129:32945)
0810 01:51:04.484217 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:04.484286 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":132,"threads_started":1}
I20260810 01:51:04.701692 3529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:04.701081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.130:40377)
0810 01:51:04.701467 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:04.701548 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":126,"threads_started":1}
I20260810 01:51:04.922786 3531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:04.922235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.131:52591)
0810 01:51:04.922565 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:04.922639 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":111,"threads_started":1}
I20260810 01:51:05.486254 3532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:05.485660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.129:34953)
0810 01:51:05.486054 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:05.486120 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":147,"threads_started":1}
I20260810 01:51:05.703673 3535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:05.703098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.130:37559)
0810 01:51:05.703460 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:05.703537 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":129,"threads_started":1}
I20260810 01:51:05.898968 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 2823
I20260810 01:51:05.924808 3536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:05.924261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37329 (local address 127.9.77.131:50231)
0810 01:51:05.924590 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:05.924671 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.190:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":115,"threads_started":1}
I20260810 01:51:05.926744 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 3072
I20260810 01:51:05.945864 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 3268
2026-08-10T01:51:05Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (60477 ms)
[----------] 30 tests from SecurityITest (273074 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:51:06 dist-test-slave-kvfs krb5kdc[3543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:51:06 dist-test-slave-kvfs krb5kdc[3543](info): set up 2 sockets
Aug 10 01:51:06 dist-test-slave-kvfs krb5kdc[3543](info): commencing operation
krb5kdc: starting...
W20260810 01:51:07.267560 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.282s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:51:07 dist-test-slave-kvfs krb5kdc[3543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:51:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:51:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:51:07.417127 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:40245
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:40245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:07.561384 3564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:07.561803 3564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:07.561889 3564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:07.566103 3564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:07.566203 3564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:07.566246 3564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:07.566286 3564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:07.566345 3564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:07.572176 3564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:40245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:40245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:07.573664 3564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:07.574988 3564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 01:51:07.585843 3564 server_base.cc:1034] running on GCE node
W20260810 01:51:07.585772 3570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:07.585907 3576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:07.585774 3572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:07.586897 3564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:07.588446 3564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:07.589663 3564 hybrid_clock.cc:648] HybridClock initialized: now 1786326667589638 us; error 42 us; skew 500 ppm
Aug 10 01:51:07 dist-test-slave-kvfs krb5kdc[3543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326667, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:07.593251 3564 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:51:07.594364 3564 webserver.cc:460] Webserver started at http://127.9.77.190:40415/ using document root <none> and password file <none>
I20260810 01:51:07.595117 3564 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:07.595219 3564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:07.595505 3564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:07.598215 3564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "004797e1cd3d456ab7e88cb81ebb6d93"
format_stamp: "Formatted at 2026-08-10 01:51:07 on dist-test-slave-kvfs"
server_key: "149acfeaa99e902bd57ca90c8b32f626"
server_key_iv: "0ccecd38e698d792b9300a015ac35129"
server_key_version: "encryptionkey@0"
I20260810 01:51:07.598917 3564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "004797e1cd3d456ab7e88cb81ebb6d93"
format_stamp: "Formatted at 2026-08-10 01:51:07 on dist-test-slave-kvfs"
server_key: "149acfeaa99e902bd57ca90c8b32f626"
server_key_iv: "0ccecd38e698d792b9300a015ac35129"
server_key_version: "encryptionkey@0"
I20260810 01:51:07.603727 3564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 01:51:07.606803 3586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:07.608355 3564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:51:07.608513 3564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "004797e1cd3d456ab7e88cb81ebb6d93"
format_stamp: "Formatted at 2026-08-10 01:51:07 on dist-test-slave-kvfs"
server_key: "149acfeaa99e902bd57ca90c8b32f626"
server_key_iv: "0ccecd38e698d792b9300a015ac35129"
server_key_version: "encryptionkey@0"
I20260810 01:51:07.608650 3564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:07.627012 3564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:07.631081 3564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:07.631328 3564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:07.640515 3564 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:40245
I20260810 01:51:07.640551 3661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:40245 every 8 connection(s)
I20260810 01:51:07.641724 3564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:51:07.645483 3662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:07.646860 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 3564
I20260810 01:51:07.647049 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:51:07.647464 9526 external_mini_cluster.cc:1428] Setting key 3eb0e5c083b4ba01ff568326a118dc0c
I20260810 01:51:07.653865 3662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 004797e1cd3d456ab7e88cb81ebb6d93: Bootstrap starting.
I20260810 01:51:07.657436 3662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 004797e1cd3d456ab7e88cb81ebb6d93: Neither blocks nor log segments found. Creating new log.
Aug 10 01:51:07 dist-test-slave-kvfs krb5kdc[3543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:51:07.658501 3662 log.cc:824] T 00000000000000000000000000000000 P 004797e1cd3d456ab7e88cb81ebb6d93: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:07.660967 3662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 004797e1cd3d456ab7e88cb81ebb6d93: No bootstrap required, opened a new log
I20260810 01:51:07.664618 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:07.649340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42630 (local address 127.9.77.190:40245)
0810 01:51:07.650067 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:07.650087 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:07.650133 (+ 46us) server_negotiation.cc:407] Connection header received
0810 01:51:07.650908 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:07.650931 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:07.651371 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:07.651724 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:07.652880 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:07.654023 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:07.655274 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:07.655812 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:07.655958 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:07.659179 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:07.659185 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:07.659203 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:07.659235 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:07.661298 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:07.661794 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:07.661800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:07.661805 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:07.661932 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:07.662239 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:07.662243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:07.662245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:07.662715 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:07.662948 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:07.663266 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 01:51:07.663628 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":200,"threads_started":1}
I20260810 01:51:07.665293 3662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 004797e1cd3d456ab7e88cb81ebb6d93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004797e1cd3d456ab7e88cb81ebb6d93" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 40245 } }
I20260810 01:51:07.665583 3662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 004797e1cd3d456ab7e88cb81ebb6d93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:07.665673 3662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 004797e1cd3d456ab7e88cb81ebb6d93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 004797e1cd3d456ab7e88cb81ebb6d93, State: Initialized, Role: FOLLOWER
I20260810 01:51:07.666276 3662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 004797e1cd3d456ab7e88cb81ebb6d93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004797e1cd3d456ab7e88cb81ebb6d93" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 40245 } }
I20260810 01:51:07.666463 3662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 004797e1cd3d456ab7e88cb81ebb6d93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:51:07.666536 3662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 004797e1cd3d456ab7e88cb81ebb6d93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:51:07.666640 3662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 004797e1cd3d456ab7e88cb81ebb6d93 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:07.667937 3662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 004797e1cd3d456ab7e88cb81ebb6d93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004797e1cd3d456ab7e88cb81ebb6d93" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 40245 } }
I20260810 01:51:07.668357 3662 leader_election.cc:302] T 00000000000000000000000000000000 P 004797e1cd3d456ab7e88cb81ebb6d93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 004797e1cd3d456ab7e88cb81ebb6d93; no voters:
I20260810 01:51:07.668673 3662 leader_election.cc:288] T 00000000000000000000000000000000 P 004797e1cd3d456ab7e88cb81ebb6d93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:51:07.668834 3672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 004797e1cd3d456ab7e88cb81ebb6d93 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:07.669111 3672 raft_consensus.cc:686] T 00000000000000000000000000000000 P 004797e1cd3d456ab7e88cb81ebb6d93 [term 1 LEADER]: Becoming Leader. State: Replica: 004797e1cd3d456ab7e88cb81ebb6d93, State: Running, Role: LEADER
I20260810 01:51:07.669516 3672 consensus_queue.cc:237] T 00000000000000000000000000000000 P 004797e1cd3d456ab7e88cb81ebb6d93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004797e1cd3d456ab7e88cb81ebb6d93" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 40245 } }
I20260810 01:51:07.670176 3662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 004797e1cd3d456ab7e88cb81ebb6d93 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:07.671675 3674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 004797e1cd3d456ab7e88cb81ebb6d93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 004797e1cd3d456ab7e88cb81ebb6d93. Latest consensus state: current_term: 1 leader_uuid: "004797e1cd3d456ab7e88cb81ebb6d93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004797e1cd3d456ab7e88cb81ebb6d93" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 40245 } } }
I20260810 01:51:07.671868 3674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 004797e1cd3d456ab7e88cb81ebb6d93 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:07.672232 3673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 004797e1cd3d456ab7e88cb81ebb6d93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "004797e1cd3d456ab7e88cb81ebb6d93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "004797e1cd3d456ab7e88cb81ebb6d93" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 40245 } } }
I20260810 01:51:07.672369 3673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 004797e1cd3d456ab7e88cb81ebb6d93 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:07.672442 3688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:51:07.676642 3688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:51:07.683763 3688 catalog_manager.cc:1347] Generated new cluster ID: 9cceb556e6f5423baad0d1a8df6d6542
I20260810 01:51:07.683852 3688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:51:07.703933 3688 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:51:07.705351 3688 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:51:07.713104 3688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 004797e1cd3d456ab7e88cb81ebb6d93: Generated new TSK 0
I20260810 01:51:07.714139 3688 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.77.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
I20260810 01:51:07.756572 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:0
--local_ip_for_outbound_sockets=127.9.77.129
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:40245
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:07.894888 3703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:07.895357 3703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:07.895406 3703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:07.899652 3703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:07.899735 3703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:07.899837 3703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:51:07.906075 3703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:40245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:07.907629 3703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:07.909001 3703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:07.921041 3709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:07.921041 3710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:07.921161 3714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:07.921548 3703 server_base.cc:1034] running on GCE node
I20260810 01:51:07.922111 3703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:07.922955 3703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:07.924175 3703 hybrid_clock.cc:648] HybridClock initialized: now 1786326667924135 us; error 60 us; skew 500 ppm
Aug 10 01:51:07 dist-test-slave-kvfs krb5kdc[3543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326667, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:07.927486 3703 init.cc:377] Logged in from keytab as kudu/127.9.77.129@KRBTEST.COM (short username kudu)
I20260810 01:51:07.928655 3703 webserver.cc:460] Webserver started at http://127.9.77.129:45117/ using document root <none> and password file <none>
I20260810 01:51:07.929371 3703 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:07.929471 3703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:07.929764 3703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:07.932453 3703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-0/data/instance:
uuid: "57d2456d86a94f10ad33cd85e64abb29"
format_stamp: "Formatted at 2026-08-10 01:51:07 on dist-test-slave-kvfs"
server_key: "139ecdf47ddcb51937611d9687d7979f"
server_key_iv: "78245f92189fe83799661edb87a06273"
server_key_version: "encryptionkey@0"
I20260810 01:51:07.933171 3703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance:
uuid: "57d2456d86a94f10ad33cd85e64abb29"
format_stamp: "Formatted at 2026-08-10 01:51:07 on dist-test-slave-kvfs"
server_key: "139ecdf47ddcb51937611d9687d7979f"
server_key_iv: "78245f92189fe83799661edb87a06273"
server_key_version: "encryptionkey@0"
I20260810 01:51:07.937949 3703 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 01:51:07.941291 3727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:07.942868 3703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:51:07.943040 3703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "57d2456d86a94f10ad33cd85e64abb29"
format_stamp: "Formatted at 2026-08-10 01:51:07 on dist-test-slave-kvfs"
server_key: "139ecdf47ddcb51937611d9687d7979f"
server_key_iv: "78245f92189fe83799661edb87a06273"
server_key_version: "encryptionkey@0"
I20260810 01:51:07.943188 3703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:07.963759 3703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:07.967471 3703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:07.967727 3703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:07.968520 3703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:51:07.969568 3703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:51:07.969655 3703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:07.969815 3703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:51:07.969864 3703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:07.980782 3703 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:33021
I20260810 01:51:07.980808 3903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:33021 every 8 connection(s)
I20260810 01:51:07.981912 3703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:51:07.987038 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 3703
I20260810 01:51:07.987223 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance
I20260810 01:51:07.987681 9526 external_mini_cluster.cc:1428] Setting key 39b4e7de57f69f331d4b37bcadfdbdb5
Aug 10 01:51:07 dist-test-slave-kvfs krb5kdc[3543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326667, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:51:08.000360 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:07.984340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:49931 (local address 127.9.77.190:40245)
0810 01:51:07.984533 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:07.984538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:07.985650 (+ 1112us) server_negotiation.cc:407] Connection header received
0810 01:51:07.987623 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:07.987627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:07.987695 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:07.987804 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:07.989637 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:07.990498 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:07.991823 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:07.992291 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:07.992421 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:07.996014 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:07.996061 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:07.996067 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:07.996107 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:07.997883 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:07.998503 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:07.998507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:07.998509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:07.998603 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:07.998982 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:07.998987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:07.998989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:07.999216 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:07.999333 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:07.999953 (+ 620us) server_negotiation.cc:299] Negotiation successful
0810 01:51:08.000154 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
WARNING: no policy specified for kudu/127.9.77.130@KRBTEST.COM; defaulting to no policy
I20260810 01:51:08.001551 3906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:07.984875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:40245 (local address 127.9.77.129:49931)
0810 01:51:07.985468 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:07.985521 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:07.987271 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:07.988121 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:07.988131 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:07.988703 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:07.989414 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:07.989438 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:07.990647 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:07.990650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:07.991688 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:07.991698 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:07.992515 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:07.992523 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:07.992805 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:07.993674 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:51:07.993706 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:07.995800 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:07.998023 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:07.998031 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:07.998054 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:07.998388 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:07.998705 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:07.998709 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:07.998711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:07.998883 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:07.999330 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:07.999339 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:07.999680 (+ 341us) client_negotiation.cc:769] Sending connection context
0810 01:51:07.999912 (+ 232us) client_negotiation.cc:240] Negotiation successful
0810 01:51:08.000377 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":177,"threads_started":1}
I20260810 01:51:08.003585 3904 heartbeater.cc:347] Connected to a master server at 127.9.77.190:40245
I20260810 01:51:08.003964 3904 heartbeater.cc:464] Registering TS with master...
I20260810 01:51:08.004710 3904 heartbeater.cc:511] Master 127.9.77.190:40245 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.130@KRBTEST.COM" created.
I20260810 01:51:08.006834 3609 ts_manager.cc:194] Registered new tserver with Master: 57d2456d86a94f10ad33cd85e64abb29 (127.9.77.129:33021)
I20260810 01:51:08.008385 3609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.129@KRBTEST.COM'} at 127.9.77.129:49931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
I20260810 01:51:08.028296 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:0
--local_ip_for_outbound_sockets=127.9.77.130
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:40245
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:08.166779 3910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:08.167217 3910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:08.167318 3910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:08.172099 3910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:08.172219 3910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:08.172340 3910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:51:08.178454 3910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:40245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:08.179978 3910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:08.181411 3910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:08.193881 3919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:08.194022 3910 server_base.cc:1034] running on GCE node
W20260810 01:51:08.193907 3915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:08.193885 3923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:08.194759 3910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:08.195662 3910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:08.196859 3910 hybrid_clock.cc:648] HybridClock initialized: now 1786326668196821 us; error 54 us; skew 500 ppm
Aug 10 01:51:08 dist-test-slave-kvfs krb5kdc[3543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326668, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:08.200281 3910 init.cc:377] Logged in from keytab as kudu/127.9.77.130@KRBTEST.COM (short username kudu)
I20260810 01:51:08.201426 3910 webserver.cc:460] Webserver started at http://127.9.77.130:40451/ using document root <none> and password file <none>
I20260810 01:51:08.202153 3910 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:08.202250 3910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:08.202533 3910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:08.205190 3910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-1/data/instance:
uuid: "e3ff27df1f8c4a5d9b04cef90087b3f3"
format_stamp: "Formatted at 2026-08-10 01:51:08 on dist-test-slave-kvfs"
server_key: "5386486dbdba74d5955cef4d8a8b5af2"
server_key_iv: "8e64533322507f4acaa2742963ac0183"
server_key_version: "encryptionkey@0"
I20260810 01:51:08.205874 3910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance:
uuid: "e3ff27df1f8c4a5d9b04cef90087b3f3"
format_stamp: "Formatted at 2026-08-10 01:51:08 on dist-test-slave-kvfs"
server_key: "5386486dbdba74d5955cef4d8a8b5af2"
server_key_iv: "8e64533322507f4acaa2742963ac0183"
server_key_version: "encryptionkey@0"
I20260810 01:51:08.210568 3910 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 01:51:08.213826 3931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:08.215595 3910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 01:51:08.215757 3910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "e3ff27df1f8c4a5d9b04cef90087b3f3"
format_stamp: "Formatted at 2026-08-10 01:51:08 on dist-test-slave-kvfs"
server_key: "5386486dbdba74d5955cef4d8a8b5af2"
server_key_iv: "8e64533322507f4acaa2742963ac0183"
server_key_version: "encryptionkey@0"
I20260810 01:51:08.215903 3910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:08.239089 3910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:08.243467 3910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:08.243722 3910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:08.244575 3910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:51:08.245649 3910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:51:08.245743 3910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:08.245823 3910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:51:08.245872 3910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:08.256616 3910 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:40393
I20260810 01:51:08.256641 4091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:40393 every 8 connection(s)
I20260810 01:51:08.257746 3910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:51:08.258556 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 3910
I20260810 01:51:08.258709 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance
I20260810 01:51:08.259094 9526 external_mini_cluster.cc:1428] Setting key 79ac624797905effbf76c567a0a170d8
Aug 10 01:51:08 dist-test-slave-kvfs krb5kdc[3543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326668, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.77.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.131@KRBTEST.COM" created.
I20260810 01:51:08.276834 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:08.260165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:60217 (local address 127.9.77.190:40245)
0810 01:51:08.260328 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:08.260332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:08.262235 (+ 1903us) server_negotiation.cc:407] Connection header received
0810 01:51:08.263860 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:08.263866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:08.263943 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:08.264077 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:08.265544 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.266471 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.268499 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.268793 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.268924 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:08.272447 (+ 3523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:08.272451 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:08.272454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:08.272494 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:08.274240 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:08.274857 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:08.274863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:08.274870 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:08.274981 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:08.275384 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:08.275392 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:08.275393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:08.275626 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:08.275747 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:08.276450 (+ 703us) server_negotiation.cc:299] Negotiation successful
0810 01:51:08.276626 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 01:51:08.277948 4094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:08.260815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:40245 (local address 127.9.77.130:60217)
0810 01:51:08.262061 (+ 1246us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:08.262109 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:08.263560 (+ 1451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:08.264300 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:08.264308 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:08.264748 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:08.265366 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:08.265382 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.266666 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.266705 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:08.268330 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:08.268342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.268989 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.268994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:08.269282 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:08.270220 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:51:08.270249 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:08.272244 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:08.274393 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:08.274400 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:08.274421 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:08.274736 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:08.275097 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:08.275100 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:08.275103 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:08.275270 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:08.275744 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:08.275753 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:08.276144 (+ 391us) client_negotiation.cc:769] Sending connection context
0810 01:51:08.276373 (+ 229us) client_negotiation.cc:240] Negotiation successful
0810 01:51:08.276810 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"thread_start_us":181,"threads_started":1}
I20260810 01:51:08.279688 4092 heartbeater.cc:347] Connected to a master server at 127.9.77.190:40245
I20260810 01:51:08.280079 4092 heartbeater.cc:464] Registering TS with master...
I20260810 01:51:08.280752 4092 heartbeater.cc:511] Master 127.9.77.190:40245 requested a full tablet report, sending...
I20260810 01:51:08.282128 3609 ts_manager.cc:194] Registered new tserver with Master: e3ff27df1f8c4a5d9b04cef90087b3f3 (127.9.77.130:40393)
I20260810 01:51:08.282838 3609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.130@KRBTEST.COM'} at 127.9.77.130:60217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
I20260810 01:51:08.297399 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:0
--local_ip_for_outbound_sockets=127.9.77.131
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:40245
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:08.435928 4100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:08.436407 4100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:08.436503 4100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:08.440948 4100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:08.441092 4100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:08.441203 4100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:51:08.447636 4100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:40245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:08.449307 4100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:08.450744 4100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:08.462983 4110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:08.463024 4109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:08.463161 4100 server_base.cc:1034] running on GCE node
W20260810 01:51:08.463012 4115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:08.463876 4100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:08.464722 4100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:08.465924 4100 hybrid_clock.cc:648] HybridClock initialized: now 1786326668465892 us; error 53 us; skew 500 ppm
Aug 10 01:51:08 dist-test-slave-kvfs krb5kdc[3543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326668, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:08.469342 4100 init.cc:377] Logged in from keytab as kudu/127.9.77.131@KRBTEST.COM (short username kudu)
I20260810 01:51:08.470520 4100 webserver.cc:460] Webserver started at http://127.9.77.131:43511/ using document root <none> and password file <none>
I20260810 01:51:08.471339 4100 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:08.471446 4100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:08.471791 4100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:08.474608 4100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-2/data/instance:
uuid: "049e3f34bcbf4597a80ad0e03fa90a78"
format_stamp: "Formatted at 2026-08-10 01:51:08 on dist-test-slave-kvfs"
server_key: "f3e0b777ec8c7cdda0407255c96feacf"
server_key_iv: "e4321dec67ad49dec2265eee7479b5df"
server_key_version: "encryptionkey@0"
I20260810 01:51:08.475369 4100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance:
uuid: "049e3f34bcbf4597a80ad0e03fa90a78"
format_stamp: "Formatted at 2026-08-10 01:51:08 on dist-test-slave-kvfs"
server_key: "f3e0b777ec8c7cdda0407255c96feacf"
server_key_iv: "e4321dec67ad49dec2265eee7479b5df"
server_key_version: "encryptionkey@0"
I20260810 01:51:08.480422 4100 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 01:51:08.483472 4124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:08.485200 4100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:51:08.485373 4100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "049e3f34bcbf4597a80ad0e03fa90a78"
format_stamp: "Formatted at 2026-08-10 01:51:08 on dist-test-slave-kvfs"
server_key: "f3e0b777ec8c7cdda0407255c96feacf"
server_key_iv: "e4321dec67ad49dec2265eee7479b5df"
server_key_version: "encryptionkey@0"
I20260810 01:51:08.485515 4100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:08.511371 4100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:08.515986 4100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:08.516360 4100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:08.517194 4100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:51:08.518373 4100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:51:08.518431 4100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:08.518517 4100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:51:08.518563 4100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:08.530726 4100 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:44945
I20260810 01:51:08.530759 4291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:44945 every 8 connection(s)
I20260810 01:51:08.531975 4100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:51:08.537503 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 4100
I20260810 01:51:08.537686 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance
I20260810 01:51:08.538080 9526 external_mini_cluster.cc:1428] Setting key d9ca9d5dc6a656f78a6a587fe345c0e5
Aug 10 01:51:08 dist-test-slave-kvfs krb5kdc[3543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326668, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:51:08.549490 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:08.534478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:54765 (local address 127.9.77.190:40245)
0810 01:51:08.534686 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:08.534691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:08.535779 (+ 1088us) server_negotiation.cc:407] Connection header received
0810 01:51:08.537574 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:08.537580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:08.537668 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:08.537777 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:08.539612 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.540495 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.541881 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.542125 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.542217 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:08.545148 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:08.545151 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:08.545154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:08.545186 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:08.546833 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:08.547592 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:08.547596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:08.547597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:08.547695 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:08.548133 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:08.548137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:08.548138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:08.548448 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:08.548549 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:08.549191 (+ 642us) server_negotiation.cc:299] Negotiation successful
0810 01:51:08.549323 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 01:51:08.550750 4295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:08.534967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:40245 (local address 127.9.77.131:54765)
0810 01:51:08.535586 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:08.535642 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:08.537295 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:08.538037 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:08.538046 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:08.538553 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:08.539420 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:08.539458 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.540692 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.540696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:08.541757 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:08.541768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.542281 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.542283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:08.542467 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:08.543158 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:51:08.543177 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:08.544965 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:08.546969 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:08.546976 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:08.546995 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:08.547412 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:08.547825 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:08.547827 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:08.547829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:08.548003 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:08.548575 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:08.548583 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:08.548922 (+ 339us) client_negotiation.cc:769] Sending connection context
0810 01:51:08.549122 (+ 200us) client_negotiation.cc:240] Negotiation successful
0810 01:51:08.549494 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":155,"threads_started":1}
I20260810 01:51:08.552397 4292 heartbeater.cc:347] Connected to a master server at 127.9.77.190:40245
I20260810 01:51:08.552726 4292 heartbeater.cc:464] Registering TS with master...
I20260810 01:51:08.553385 4292 heartbeater.cc:511] Master 127.9.77.190:40245 requested a full tablet report, sending...
I20260810 01:51:08.554644 3608 ts_manager.cc:194] Registered new tserver with Master: 049e3f34bcbf4597a80ad0e03fa90a78 (127.9.77.131:44945)
I20260810 01:51:08.555323 3608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.131@KRBTEST.COM'} at 127.9.77.131:54765
I20260810 01:51:08.565344 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 01:51:08.576825 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:08.567842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42646 (local address 127.9.77.190:40245)
0810 01:51:08.568051 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:08.568057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:08.568294 (+ 237us) server_negotiation.cc:407] Connection header received
0810 01:51:08.568496 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:08.568500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:08.568562 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:08.568652 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:08.569516 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.570177 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.571193 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.571457 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.571570 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:08.573307 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:08.573311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:08.573314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:08.573353 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:08.574757 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:08.575256 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:08.575260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:08.575262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:08.575363 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:08.575677 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:08.575680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:08.575682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:08.575984 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:08.576206 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:08.576475 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 01:51:08.576618 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 01:51:08.580681 3608 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 01:51:08.584774 3608 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:51:08.602900 4310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:08.596111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:33021 (local address 127.0.0.1:38402)
0810 01:51:08.597048 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:08.597127 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:08.597305 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:08.598152 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:08.598158 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:08.598178 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:08.598366 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:08.598374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.599848 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.599851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:08.601083 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:08.601093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.602195 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.602200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:08.602326 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:08.602377 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 01:51:08.602521 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 01:51:08.602680 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":83,"thread_start_us":118,"threads_started":1}
I20260810 01:51:08.602908 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:08.596341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38402 (local address 127.9.77.129:33021)
0810 01:51:08.597338 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:08.597346 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:08.597382 (+ 36us) server_negotiation.cc:407] Connection header received
0810 01:51:08.597498 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:08.597505 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:08.597787 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:08.597972 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:08.598529 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.599671 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.601214 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.602052 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.602172 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:08.602375 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:08.602575 (+ 200us) server_negotiation.cc:299] Negotiation successful
0810 01:51:08.602701 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":868,"thread_start_us":180,"threads_started":1}
I20260810 01:51:08.603688 4314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:08.597119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:40393 (local address 127.0.0.1:60050)
0810 01:51:08.597697 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:08.597712 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:08.597807 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:08.598743 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:08.598747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:08.598767 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:08.598877 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:08.598885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.600590 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.600594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:08.601715 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:08.601723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.603234 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.603241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:08.603346 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:08.603373 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 01:51:08.603456 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 01:51:08.603536 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":2,"thread_start_us":106,"threads_started":1}
I20260810 01:51:08.603962 4315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:08.597417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60050 (local address 127.9.77.130:40393)
0810 01:51:08.598043 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:08.598058 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:08.598082 (+ 24us) server_negotiation.cc:407] Connection header received
0810 01:51:08.598170 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:08.598177 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:08.598379 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:08.598537 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:08.599030 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.600440 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.601843 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.602686 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.602802 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:08.602955 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:08.603629 (+ 674us) server_negotiation.cc:299] Negotiation successful
0810 01:51:08.603766 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":226,"threads_started":1}
I20260810 01:51:08.604790 4313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:08.596460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34348 (local address 127.9.77.131:44945)
0810 01:51:08.597032 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:08.597048 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:08.597794 (+ 746us) server_negotiation.cc:407] Connection header received
0810 01:51:08.597907 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:08.597912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:08.598125 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:08.598308 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:08.598532 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.599513 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.601576 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.602466 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.602597 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:08.602739 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:08.603406 (+ 667us) server_negotiation.cc:299] Negotiation successful
0810 01:51:08.604647 (+ 1241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":198,"threads_started":1}
I20260810 01:51:08.603294 4311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:08.596282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:44945 (local address 127.0.0.1:34348)
0810 01:51:08.597523 (+ 1241us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:08.597539 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:08.597655 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:08.598285 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:08.598288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:08.598301 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:08.598409 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:08.598418 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.599664 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.599671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:08.601428 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:08.601440 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.602816 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.602820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:08.602914 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:08.602953 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 01:51:08.603034 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 01:51:08.603119 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":109,"thread_start_us":139,"threads_started":1}
I20260810 01:51:08.607280 3998 tablet_service.cc:1467] Processing CreateTablet for tablet 5b61496711f2452e8e3926b5ab5c00c4 (DEFAULT_TABLE table=test-table [id=e7d32814211044f6b88998b124e17548]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:51:08.607283 3799 tablet_service.cc:1467] Processing CreateTablet for tablet 5b61496711f2452e8e3926b5ab5c00c4 (DEFAULT_TABLE table=test-table [id=e7d32814211044f6b88998b124e17548]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:51:08.607813 4199 tablet_service.cc:1467] Processing CreateTablet for tablet 5b61496711f2452e8e3926b5ab5c00c4 (DEFAULT_TABLE table=test-table [id=e7d32814211044f6b88998b124e17548]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:51:08.608783 3799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b61496711f2452e8e3926b5ab5c00c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:08.608789 4199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b61496711f2452e8e3926b5ab5c00c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:08.608786 3998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b61496711f2452e8e3926b5ab5c00c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:08.617774 4319 tablet_bootstrap.cc:492] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29: Bootstrap starting.
I20260810 01:51:08.618449 4320 tablet_bootstrap.cc:492] T 5b61496711f2452e8e3926b5ab5c00c4 P 049e3f34bcbf4597a80ad0e03fa90a78: Bootstrap starting.
I20260810 01:51:08.620067 4321 tablet_bootstrap.cc:492] T 5b61496711f2452e8e3926b5ab5c00c4 P e3ff27df1f8c4a5d9b04cef90087b3f3: Bootstrap starting.
I20260810 01:51:08.621095 4319 tablet_bootstrap.cc:654] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:08.621095 4320 tablet_bootstrap.cc:654] T 5b61496711f2452e8e3926b5ab5c00c4 P 049e3f34bcbf4597a80ad0e03fa90a78: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:08.622303 4319 log.cc:824] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:08.622378 4321 tablet_bootstrap.cc:654] T 5b61496711f2452e8e3926b5ab5c00c4 P e3ff27df1f8c4a5d9b04cef90087b3f3: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:08.622619 4320 log.cc:824] T 5b61496711f2452e8e3926b5ab5c00c4 P 049e3f34bcbf4597a80ad0e03fa90a78: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:08.623790 4321 log.cc:824] T 5b61496711f2452e8e3926b5ab5c00c4 P e3ff27df1f8c4a5d9b04cef90087b3f3: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:08.625034 4319 tablet_bootstrap.cc:492] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29: No bootstrap required, opened a new log
I20260810 01:51:08.625097 4320 tablet_bootstrap.cc:492] T 5b61496711f2452e8e3926b5ab5c00c4 P 049e3f34bcbf4597a80ad0e03fa90a78: No bootstrap required, opened a new log
I20260810 01:51:08.625274 4319 ts_tablet_manager.cc:1397] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 01:51:08.625295 4320 ts_tablet_manager.cc:1397] T 5b61496711f2452e8e3926b5ab5c00c4 P 049e3f34bcbf4597a80ad0e03fa90a78: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 01:51:08.626652 4321 tablet_bootstrap.cc:492] T 5b61496711f2452e8e3926b5ab5c00c4 P e3ff27df1f8c4a5d9b04cef90087b3f3: No bootstrap required, opened a new log
I20260810 01:51:08.626926 4321 ts_tablet_manager.cc:1397] T 5b61496711f2452e8e3926b5ab5c00c4 P e3ff27df1f8c4a5d9b04cef90087b3f3: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260810 01:51:08.629187 4320 raft_consensus.cc:348] T 5b61496711f2452e8e3926b5ab5c00c4 P 049e3f34bcbf4597a80ad0e03fa90a78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d2456d86a94f10ad33cd85e64abb29" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 33021 } } peers { permanent_uuid: "049e3f34bcbf4597a80ad0e03fa90a78" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 44945 } } peers { permanent_uuid: "e3ff27df1f8c4a5d9b04cef90087b3f3" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40393 } }
I20260810 01:51:08.629508 4320 raft_consensus.cc:374] T 5b61496711f2452e8e3926b5ab5c00c4 P 049e3f34bcbf4597a80ad0e03fa90a78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:08.629604 4320 raft_consensus.cc:729] T 5b61496711f2452e8e3926b5ab5c00c4 P 049e3f34bcbf4597a80ad0e03fa90a78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 049e3f34bcbf4597a80ad0e03fa90a78, State: Initialized, Role: FOLLOWER
I20260810 01:51:08.630203 4320 consensus_queue.cc:260] T 5b61496711f2452e8e3926b5ab5c00c4 P 049e3f34bcbf4597a80ad0e03fa90a78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d2456d86a94f10ad33cd85e64abb29" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 33021 } } peers { permanent_uuid: "049e3f34bcbf4597a80ad0e03fa90a78" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 44945 } } peers { permanent_uuid: "e3ff27df1f8c4a5d9b04cef90087b3f3" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40393 } }
I20260810 01:51:08.630461 4319 raft_consensus.cc:348] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d2456d86a94f10ad33cd85e64abb29" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 33021 } } peers { permanent_uuid: "049e3f34bcbf4597a80ad0e03fa90a78" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 44945 } } peers { permanent_uuid: "e3ff27df1f8c4a5d9b04cef90087b3f3" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40393 } }
I20260810 01:51:08.630816 4319 raft_consensus.cc:374] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:08.630913 4319 raft_consensus.cc:729] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57d2456d86a94f10ad33cd85e64abb29, State: Initialized, Role: FOLLOWER
I20260810 01:51:08.631028 4292 heartbeater.cc:503] Master 127.9.77.190:40245 was elected leader, sending a full tablet report...
I20260810 01:51:08.631112 4321 raft_consensus.cc:348] T 5b61496711f2452e8e3926b5ab5c00c4 P e3ff27df1f8c4a5d9b04cef90087b3f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d2456d86a94f10ad33cd85e64abb29" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 33021 } } peers { permanent_uuid: "049e3f34bcbf4597a80ad0e03fa90a78" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 44945 } } peers { permanent_uuid: "e3ff27df1f8c4a5d9b04cef90087b3f3" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40393 } }
I20260810 01:51:08.631392 4321 raft_consensus.cc:374] T 5b61496711f2452e8e3926b5ab5c00c4 P e3ff27df1f8c4a5d9b04cef90087b3f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:08.631408 4319 consensus_queue.cc:260] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d2456d86a94f10ad33cd85e64abb29" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 33021 } } peers { permanent_uuid: "049e3f34bcbf4597a80ad0e03fa90a78" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 44945 } } peers { permanent_uuid: "e3ff27df1f8c4a5d9b04cef90087b3f3" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40393 } }
I20260810 01:51:08.631493 4321 raft_consensus.cc:729] T 5b61496711f2452e8e3926b5ab5c00c4 P e3ff27df1f8c4a5d9b04cef90087b3f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3ff27df1f8c4a5d9b04cef90087b3f3, State: Initialized, Role: FOLLOWER
I20260810 01:51:08.631922 4321 consensus_queue.cc:260] T 5b61496711f2452e8e3926b5ab5c00c4 P e3ff27df1f8c4a5d9b04cef90087b3f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d2456d86a94f10ad33cd85e64abb29" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 33021 } } peers { permanent_uuid: "049e3f34bcbf4597a80ad0e03fa90a78" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 44945 } } peers { permanent_uuid: "e3ff27df1f8c4a5d9b04cef90087b3f3" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40393 } }
I20260810 01:51:08.632638 4092 heartbeater.cc:503] Master 127.9.77.190:40245 was elected leader, sending a full tablet report...
I20260810 01:51:08.632934 4321 ts_tablet_manager.cc:1428] T 5b61496711f2452e8e3926b5ab5c00c4 P e3ff27df1f8c4a5d9b04cef90087b3f3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 01:51:08.633764 4319 ts_tablet_manager.cc:1428] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260810 01:51:08.634488 3904 heartbeater.cc:503] Master 127.9.77.190:40245 was elected leader, sending a full tablet report...
I20260810 01:51:08.635921 4320 ts_tablet_manager.cc:1428] T 5b61496711f2452e8e3926b5ab5c00c4 P 049e3f34bcbf4597a80ad0e03fa90a78: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260810 01:51:08.697166 4333 raft_consensus.cc:482] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:51:08.697489 4333 raft_consensus.cc:504] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d2456d86a94f10ad33cd85e64abb29" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 33021 } } peers { permanent_uuid: "049e3f34bcbf4597a80ad0e03fa90a78" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 44945 } } peers { permanent_uuid: "e3ff27df1f8c4a5d9b04cef90087b3f3" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40393 } }
I20260810 01:51:08.698938 4333 leader_election.cc:288] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 049e3f34bcbf4597a80ad0e03fa90a78 (127.9.77.131:44945), e3ff27df1f8c4a5d9b04cef90087b3f3 (127.9.77.130:40393)
I20260810 01:51:08.703723 4315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:08.700173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:40265 (local address 127.9.77.130:40393)
0810 01:51:08.700363 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:08.700369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:08.700386 (+ 17us) server_negotiation.cc:407] Connection header received
0810 01:51:08.700506 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:08.700509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:08.700568 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:08.700650 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:08.700987 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.701514 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.702642 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.703101 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.703242 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:08.703305 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:08.703495 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 01:51:08.703577 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 01:51:08.703723 4313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:08.699411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:58597 (local address 127.9.77.131:44945)
0810 01:51:08.699587 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:08.699591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:08.699891 (+ 300us) server_negotiation.cc:407] Connection header received
0810 01:51:08.700151 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:08.700157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:08.700253 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:08.700315 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:08.700871 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.701432 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.702621 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.703080 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.703228 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:08.703301 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:08.703483 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 01:51:08.703571 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 01:51:08.703752 4335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:08.699341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:44945 (local address 127.9.77.129:58597)
0810 01:51:08.699744 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:08.699765 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:08.699913 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:08.700513 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:08.700516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:08.700537 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:08.700737 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:08.700747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.701564 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.701569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:08.702486 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:08.702495 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.703215 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.703218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:08.703318 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:08.703359 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 01:51:08.703459 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 01:51:08.703540 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":3,"thread_start_us":180,"threads_started":1}
I20260810 01:51:08.703755 4336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:08.699869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:40393 (local address 127.9.77.129:40265)
0810 01:51:08.700249 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:08.700267 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:08.700392 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:08.700693 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:08.700696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:08.700713 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:08.700828 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:08.700833 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.701642 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.701648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:08.702507 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:08.702515 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.703232 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.703236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:08.703364 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:08.703381 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 01:51:08.703476 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 01:51:08.703555 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":26,"thread_start_us":137,"threads_started":1}
I20260810 01:51:08.704602 4030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b61496711f2452e8e3926b5ab5c00c4" candidate_uuid: "57d2456d86a94f10ad33cd85e64abb29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3ff27df1f8c4a5d9b04cef90087b3f3" is_pre_election: true
I20260810 01:51:08.704610 4226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b61496711f2452e8e3926b5ab5c00c4" candidate_uuid: "57d2456d86a94f10ad33cd85e64abb29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "049e3f34bcbf4597a80ad0e03fa90a78" is_pre_election: true
I20260810 01:51:08.704996 4030 raft_consensus.cc:2454] T 5b61496711f2452e8e3926b5ab5c00c4 P e3ff27df1f8c4a5d9b04cef90087b3f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57d2456d86a94f10ad33cd85e64abb29 in term 0.
I20260810 01:51:08.705000 4226 raft_consensus.cc:2454] T 5b61496711f2452e8e3926b5ab5c00c4 P 049e3f34bcbf4597a80ad0e03fa90a78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57d2456d86a94f10ad33cd85e64abb29 in term 0.
I20260810 01:51:08.705600 3736 leader_election.cc:302] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 049e3f34bcbf4597a80ad0e03fa90a78, 57d2456d86a94f10ad33cd85e64abb29; no voters:
I20260810 01:51:08.705910 4333 raft_consensus.cc:2781] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:51:08.706030 4333 raft_consensus.cc:482] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:51:08.706089 4333 raft_consensus.cc:3037] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:08.707412 4333 raft_consensus.cc:504] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d2456d86a94f10ad33cd85e64abb29" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 33021 } } peers { permanent_uuid: "049e3f34bcbf4597a80ad0e03fa90a78" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 44945 } } peers { permanent_uuid: "e3ff27df1f8c4a5d9b04cef90087b3f3" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40393 } }
I20260810 01:51:08.707828 4333 leader_election.cc:288] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29 [CANDIDATE]: Term 1 election: Requested vote from peers 049e3f34bcbf4597a80ad0e03fa90a78 (127.9.77.131:44945), e3ff27df1f8c4a5d9b04cef90087b3f3 (127.9.77.130:40393)
I20260810 01:51:08.708323 4030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b61496711f2452e8e3926b5ab5c00c4" candidate_uuid: "57d2456d86a94f10ad33cd85e64abb29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3ff27df1f8c4a5d9b04cef90087b3f3"
I20260810 01:51:08.708335 4226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b61496711f2452e8e3926b5ab5c00c4" candidate_uuid: "57d2456d86a94f10ad33cd85e64abb29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "049e3f34bcbf4597a80ad0e03fa90a78"
I20260810 01:51:08.708454 4030 raft_consensus.cc:3037] T 5b61496711f2452e8e3926b5ab5c00c4 P e3ff27df1f8c4a5d9b04cef90087b3f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:08.708488 4226 raft_consensus.cc:3037] T 5b61496711f2452e8e3926b5ab5c00c4 P 049e3f34bcbf4597a80ad0e03fa90a78 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:08.709751 4030 raft_consensus.cc:2454] T 5b61496711f2452e8e3926b5ab5c00c4 P e3ff27df1f8c4a5d9b04cef90087b3f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57d2456d86a94f10ad33cd85e64abb29 in term 1.
I20260810 01:51:08.709776 4226 raft_consensus.cc:2454] T 5b61496711f2452e8e3926b5ab5c00c4 P 049e3f34bcbf4597a80ad0e03fa90a78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57d2456d86a94f10ad33cd85e64abb29 in term 1.
I20260810 01:51:08.710129 3736 leader_election.cc:302] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57d2456d86a94f10ad33cd85e64abb29, e3ff27df1f8c4a5d9b04cef90087b3f3; no voters:
I20260810 01:51:08.710320 4333 raft_consensus.cc:2781] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:08.710594 4333 raft_consensus.cc:686] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29 [term 1 LEADER]: Becoming Leader. State: Replica: 57d2456d86a94f10ad33cd85e64abb29, State: Running, Role: LEADER
I20260810 01:51:08.711022 4333 consensus_queue.cc:237] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d2456d86a94f10ad33cd85e64abb29" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 33021 } } peers { permanent_uuid: "049e3f34bcbf4597a80ad0e03fa90a78" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 44945 } } peers { permanent_uuid: "e3ff27df1f8c4a5d9b04cef90087b3f3" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40393 } }
I20260810 01:51:08.715227 3609 catalog_manager.cc:5495] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 57d2456d86a94f10ad33cd85e64abb29 (127.9.77.129). New cstate: current_term: 1 leader_uuid: "57d2456d86a94f10ad33cd85e64abb29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57d2456d86a94f10ad33cd85e64abb29" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 33021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "049e3f34bcbf4597a80ad0e03fa90a78" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 44945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3ff27df1f8c4a5d9b04cef90087b3f3" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40393 } health_report { overall_health: UNKNOWN } } }
I20260810 01:51:08.730289 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:08.726065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38406 (local address 127.9.77.129:33021)
0810 01:51:08.726222 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:08.726227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:08.726246 (+ 19us) server_negotiation.cc:407] Connection header received
0810 01:51:08.726378 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:08.726381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:08.726454 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:08.726585 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:51:08.726872 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.727468 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.728501 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:08.728747 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:08.728860 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:08.729227 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:51:08.729831 (+ 604us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:51:08.729987 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:08.730023 (+ 36us) server_negotiation.cc:299] Negotiation successful
0810 01:51:08.730097 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260810 01:51:08.733623 3905 tablet.cc:2369] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 01:51:08.739480 4030 raft_consensus.cc:1261] T 5b61496711f2452e8e3926b5ab5c00c4 P e3ff27df1f8c4a5d9b04cef90087b3f3 [term 1 FOLLOWER]: Refusing update from remote peer 57d2456d86a94f10ad33cd85e64abb29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:51:08.739480 4226 raft_consensus.cc:1261] T 5b61496711f2452e8e3926b5ab5c00c4 P 049e3f34bcbf4597a80ad0e03fa90a78 [term 1 FOLLOWER]: Refusing update from remote peer 57d2456d86a94f10ad33cd85e64abb29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:51:08.740557 4337 consensus_queue.cc:1035] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3ff27df1f8c4a5d9b04cef90087b3f3" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 40393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:51:08.740898 4333 consensus_queue.cc:1035] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "049e3f34bcbf4597a80ad0e03fa90a78" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 44945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:51:08.764660 3607 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:51:08.764930 3607 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:51:08.768894 3607 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 004797e1cd3d456ab7e88cb81ebb6d93: Sending DeleteTablet for 3 replicas of tablet 5b61496711f2452e8e3926b5ab5c00c4
I20260810 01:51:08.770037 3799 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b61496711f2452e8e3926b5ab5c00c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:51:08 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:38402
I20260810 01:51:08.770188 3998 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b61496711f2452e8e3926b5ab5c00c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:51:08 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:60050
I20260810 01:51:08.770177 4199 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b61496711f2452e8e3926b5ab5c00c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:51:08 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:34348
I20260810 01:51:08.770808 4360 tablet_replica.cc:330] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29: stopping tablet replica
I20260810 01:51:08.771119 4362 tablet_replica.cc:330] T 5b61496711f2452e8e3926b5ab5c00c4 P e3ff27df1f8c4a5d9b04cef90087b3f3: stopping tablet replica
I20260810 01:51:08.771313 4360 raft_consensus.cc:2229] T 5b61496711f2452e8e3926b5ab5c00c4 P 57d2456d86a94f10ad33cd85e64abb29 [term 1 LEADER]: Raft consensus shutting down.
I20260810 01:51:08.771864 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 3703
I20260810 01:51:08.771850 4362 raft_consensus.cc:2229] T 5b61496711f2452e8e3926b5ab5c00c4 P e3ff27df1f8c4a5d9b04cef90087b3f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:51:08.776067 4362 raft_consensus.cc:2258] T 5b61496711f2452e8e3926b5ab5c00c4 P e3ff27df1f8c4a5d9b04cef90087b3f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:51:08.781987 4362 ts_tablet_manager.cc:1905] T 5b61496711f2452e8e3926b5ab5c00c4 P e3ff27df1f8c4a5d9b04cef90087b3f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:51:08.787971 4361 tablet_replica.cc:330] T 5b61496711f2452e8e3926b5ab5c00c4 P 049e3f34bcbf4597a80ad0e03fa90a78: stopping tablet replica
I20260810 01:51:08.788479 4362 ts_tablet_manager.cc:1918] T 5b61496711f2452e8e3926b5ab5c00c4 P e3ff27df1f8c4a5d9b04cef90087b3f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:51:08.788651 4362 log.cc:1196] T 5b61496711f2452e8e3926b5ab5c00c4 P e3ff27df1f8c4a5d9b04cef90087b3f3: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-1/wal/wals/5b61496711f2452e8e3926b5ab5c00c4
I20260810 01:51:08.789048 4362 ts_tablet_manager.cc:1939] T 5b61496711f2452e8e3926b5ab5c00c4 P e3ff27df1f8c4a5d9b04cef90087b3f3: Deleting consensus metadata
I20260810 01:51:08.789637 4361 raft_consensus.cc:2229] T 5b61496711f2452e8e3926b5ab5c00c4 P 049e3f34bcbf4597a80ad0e03fa90a78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:51:08.790180 4361 raft_consensus.cc:2258] T 5b61496711f2452e8e3926b5ab5c00c4 P 049e3f34bcbf4597a80ad0e03fa90a78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:51:08.791296 3591 catalog_manager.cc:4841] TS e3ff27df1f8c4a5d9b04cef90087b3f3 (127.9.77.130:40393): tablet 5b61496711f2452e8e3926b5ab5c00c4 (table test-table [id=e7d32814211044f6b88998b124e17548]) successfully deleted
I20260810 01:51:08.792493 4361 ts_tablet_manager.cc:1905] T 5b61496711f2452e8e3926b5ab5c00c4 P 049e3f34bcbf4597a80ad0e03fa90a78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:51:08.795570 4361 ts_tablet_manager.cc:1918] T 5b61496711f2452e8e3926b5ab5c00c4 P 049e3f34bcbf4597a80ad0e03fa90a78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:51:08.795696 4361 log.cc:1196] T 5b61496711f2452e8e3926b5ab5c00c4 P 049e3f34bcbf4597a80ad0e03fa90a78: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786326392877349-9526-0/minicluster-data/ts-2/wal/wals/5b61496711f2452e8e3926b5ab5c00c4
I20260810 01:51:08.796242 4361 ts_tablet_manager.cc:1939] T 5b61496711f2452e8e3926b5ab5c00c4 P 049e3f34bcbf4597a80ad0e03fa90a78: Deleting consensus metadata
W20260810 01:51:08.797552 3588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.77.129:33021 (error 108)
I20260810 01:51:08.798110 3591 catalog_manager.cc:4841] TS 049e3f34bcbf4597a80ad0e03fa90a78 (127.9.77.131:44945): tablet 5b61496711f2452e8e3926b5ab5c00c4 (table test-table [id=e7d32814211044f6b88998b124e17548]) successfully deleted
I20260810 01:51:08.798836 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 3910
I20260810 01:51:08.799141 4311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:08.798640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:33021 (local address 127.0.0.1:38420)
0810 01:51:08.798922 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:08.799016 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.129:33021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
W20260810 01:51:08.805040 3588 catalog_manager.cc:4584] TS 57d2456d86a94f10ad33cd85e64abb29 (127.9.77.129:33021): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5b61496711f2452e8e3926b5ab5c00c4: Network error: Client connection negotiation failed: client connection to 127.9.77.129:33021: connect: Connection refused (error 111)
I20260810 01:51:08.816376 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 4100
I20260810 01:51:08.840945 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 3564
2026-08-10T01:51:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2888 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T01:51:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:51:08Z Disabled control of system clock
I20260810 01:51:08.887365 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:37861
--webserver_interface=127.9.77.190
--webserver_port=0
--builtin_ntp_servers=127.9.77.148:34665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:37861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 01:51:09.033393 4370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:09.033792 4370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:09.033898 4370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:09.038328 4370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:09.038411 4370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:09.038440 4370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:09.038482 4370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:09.038522 4370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:09.043462 4370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:34665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:37861
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:37861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:09.045023 4370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:09.046725 4370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:09.056986 4377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:09.057139 4370 server_base.cc:1034] running on GCE node
W20260810 01:51:09.056984 4378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:09.057341 4380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:09.058079 4370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:09.059518 4370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:09.060724 4370 hybrid_clock.cc:648] HybridClock initialized: now 1786326669060703 us; error 52 us; skew 500 ppm
I20260810 01:51:09.063127 4370 webserver.cc:460] Webserver started at http://127.9.77.190:42309/ using document root <none> and password file <none>
I20260810 01:51:09.063915 4370 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:09.064028 4370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:09.064419 4370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:09.067165 4370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "fcd5fa2e772d46cab4e8d57322d1feb0"
format_stamp: "Formatted at 2026-08-10 01:51:09 on dist-test-slave-kvfs"
server_key: "b8ba2670566aa9d4636eee9ed4088685"
server_key_iv: "3493fa9b3cea4aff0fa78334199b9be2"
server_key_version: "encryptionkey@0"
I20260810 01:51:09.067963 4370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "fcd5fa2e772d46cab4e8d57322d1feb0"
format_stamp: "Formatted at 2026-08-10 01:51:09 on dist-test-slave-kvfs"
server_key: "b8ba2670566aa9d4636eee9ed4088685"
server_key_iv: "3493fa9b3cea4aff0fa78334199b9be2"
server_key_version: "encryptionkey@0"
I20260810 01:51:09.073312 4370 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260810 01:51:09.076656 4386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:09.078327 4370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:51:09.078522 4370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "fcd5fa2e772d46cab4e8d57322d1feb0"
format_stamp: "Formatted at 2026-08-10 01:51:09 on dist-test-slave-kvfs"
server_key: "b8ba2670566aa9d4636eee9ed4088685"
server_key_iv: "3493fa9b3cea4aff0fa78334199b9be2"
server_key_version: "encryptionkey@0"
I20260810 01:51:09.078665 4370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:09.096378 4370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:09.097513 4370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:09.097734 4370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:09.106696 4370 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:37861
I20260810 01:51:09.106778 4463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:37861 every 8 connection(s)
I20260810 01:51:09.107930 4370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:51:09.113500 4464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:09.116192 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 4370
I20260810 01:51:09.116374 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:51:09.116820 9526 external_mini_cluster.cc:1428] Setting key 92900c5a7c4083fe4944c4b4fe22acaf
I20260810 01:51:09.129437 4464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcd5fa2e772d46cab4e8d57322d1feb0: Bootstrap starting.
I20260810 01:51:09.131462 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:09.119274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33390 (local address 127.9.77.190:37861)
0810 01:51:09.120107 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:09.120131 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:09.120185 (+ 54us) server_negotiation.cc:407] Connection header received
0810 01:51:09.121258 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:09.121289 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:09.121912 (+ 623us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:09.122845 (+ 933us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:09.123368 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:09.124964 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:09.126289 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:09.127085 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:09.127812 (+ 727us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:09.128485 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:09.128493 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:09.128512 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:09.129096 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:09.129176 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:09.129206 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:09.129838 (+ 632us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:09.129922 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 01:51:09.130244 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":161,"threads_started":1}
I20260810 01:51:09.132984 4464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcd5fa2e772d46cab4e8d57322d1feb0: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:09.133960 4464 log.cc:824] T 00000000000000000000000000000000 P fcd5fa2e772d46cab4e8d57322d1feb0: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:09.136509 4464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcd5fa2e772d46cab4e8d57322d1feb0: No bootstrap required, opened a new log
I20260810 01:51:09.140769 4464 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcd5fa2e772d46cab4e8d57322d1feb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd5fa2e772d46cab4e8d57322d1feb0" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37861 } }
I20260810 01:51:09.141069 4464 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcd5fa2e772d46cab4e8d57322d1feb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:09.141165 4464 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcd5fa2e772d46cab4e8d57322d1feb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd5fa2e772d46cab4e8d57322d1feb0, State: Initialized, Role: FOLLOWER
I20260810 01:51:09.141666 4464 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcd5fa2e772d46cab4e8d57322d1feb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd5fa2e772d46cab4e8d57322d1feb0" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37861 } }
I20260810 01:51:09.141839 4464 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcd5fa2e772d46cab4e8d57322d1feb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:51:09.141927 4464 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcd5fa2e772d46cab4e8d57322d1feb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:51:09.142074 4464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcd5fa2e772d46cab4e8d57322d1feb0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:09.143154 4464 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcd5fa2e772d46cab4e8d57322d1feb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd5fa2e772d46cab4e8d57322d1feb0" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37861 } }
I20260810 01:51:09.143640 4464 leader_election.cc:302] T 00000000000000000000000000000000 P fcd5fa2e772d46cab4e8d57322d1feb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcd5fa2e772d46cab4e8d57322d1feb0; no voters:
I20260810 01:51:09.144100 4464 leader_election.cc:288] T 00000000000000000000000000000000 P fcd5fa2e772d46cab4e8d57322d1feb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:51:09.144269 4469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcd5fa2e772d46cab4e8d57322d1feb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:09.144573 4469 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcd5fa2e772d46cab4e8d57322d1feb0 [term 1 LEADER]: Becoming Leader. State: Replica: fcd5fa2e772d46cab4e8d57322d1feb0, State: Running, Role: LEADER
I20260810 01:51:09.145061 4469 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcd5fa2e772d46cab4e8d57322d1feb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd5fa2e772d46cab4e8d57322d1feb0" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37861 } }
I20260810 01:51:09.145632 4464 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcd5fa2e772d46cab4e8d57322d1feb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:09.147485 4470 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcd5fa2e772d46cab4e8d57322d1feb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcd5fa2e772d46cab4e8d57322d1feb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd5fa2e772d46cab4e8d57322d1feb0" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37861 } } }
I20260810 01:51:09.147548 4471 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcd5fa2e772d46cab4e8d57322d1feb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcd5fa2e772d46cab4e8d57322d1feb0. Latest consensus state: current_term: 1 leader_uuid: "fcd5fa2e772d46cab4e8d57322d1feb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd5fa2e772d46cab4e8d57322d1feb0" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37861 } } }
I20260810 01:51:09.147646 4470 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcd5fa2e772d46cab4e8d57322d1feb0 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:09.147677 4471 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcd5fa2e772d46cab4e8d57322d1feb0 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:09.148136 4489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:51:09.152851 4489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:51:09.161098 4489 catalog_manager.cc:1347] Generated new cluster ID: 084e2b2f87af4a219bc76950563a6797
I20260810 01:51:09.161185 4489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:51:09.175132 4489 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:51:09.176689 4489 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:51:09.187762 4489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcd5fa2e772d46cab4e8d57322d1feb0: Generated new TSK 0
I20260810 01:51:09.188848 4489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:51:09.204084 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:0
--local_ip_for_outbound_sockets=127.9.77.129
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:37861
--builtin_ntp_servers=127.9.77.148:34665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 01:51:09.377111 4503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:09.377462 4503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:09.377504 4503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:09.382463 4503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:09.382567 4503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:09.382669 4503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:51:09.387955 4503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:34665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:37861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:09.389554 4503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:09.391392 4503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:09.403297 4515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:09.403297 4511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:09.403297 4510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:09.403615 4503 server_base.cc:1034] running on GCE node
I20260810 01:51:09.404325 4503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:09.405112 4503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:09.406334 4503 hybrid_clock.cc:648] HybridClock initialized: now 1786326669406301 us; error 56 us; skew 500 ppm
I20260810 01:51:09.408596 4503 webserver.cc:460] Webserver started at http://127.9.77.129:35311/ using document root <none> and password file <none>
I20260810 01:51:09.409305 4503 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:09.409405 4503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:09.409682 4503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:09.412308 4503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-0/data/instance:
uuid: "5e38ba31aafb454fab3a30af25642bd2"
format_stamp: "Formatted at 2026-08-10 01:51:09 on dist-test-slave-kvfs"
server_key: "6a66bfe1c4f3cfaf9ac64f77be30b8c7"
server_key_iv: "05abd7a77e94a9f11f22b62cf707e731"
server_key_version: "encryptionkey@0"
I20260810 01:51:09.412985 4503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance:
uuid: "5e38ba31aafb454fab3a30af25642bd2"
format_stamp: "Formatted at 2026-08-10 01:51:09 on dist-test-slave-kvfs"
server_key: "6a66bfe1c4f3cfaf9ac64f77be30b8c7"
server_key_iv: "05abd7a77e94a9f11f22b62cf707e731"
server_key_version: "encryptionkey@0"
I20260810 01:51:09.417594 4503 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 01:51:09.420760 4530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:09.422431 4503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:51:09.422585 4503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "5e38ba31aafb454fab3a30af25642bd2"
format_stamp: "Formatted at 2026-08-10 01:51:09 on dist-test-slave-kvfs"
server_key: "6a66bfe1c4f3cfaf9ac64f77be30b8c7"
server_key_iv: "05abd7a77e94a9f11f22b62cf707e731"
server_key_version: "encryptionkey@0"
I20260810 01:51:09.422726 4503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:09.444591 4503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:09.445678 4503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:09.445937 4503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:09.446877 4503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:51:09.448289 4503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:51:09.448352 4503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:09.448427 4503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:51:09.448462 4503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:09.459264 4503 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:33845
I20260810 01:51:09.459295 4709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:33845 every 8 connection(s)
I20260810 01:51:09.460551 4503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:51:09.461481 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 4503
I20260810 01:51:09.461637 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance
I20260810 01:51:09.462023 9526 external_mini_cluster.cc:1428] Setting key 404c95cbeed9e585b0ec655d941a92ed
I20260810 01:51:09.464838 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:0
--local_ip_for_outbound_sockets=127.9.77.130
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:37861
--builtin_ntp_servers=127.9.77.148:34665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 01:51:09.472281 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:09.463104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:58971 (local address 127.9.77.190:37861)
0810 01:51:09.463274 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:09.463280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:09.464848 (+ 1568us) server_negotiation.cc:407] Connection header received
0810 01:51:09.466553 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:09.466557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:09.466619 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:09.466707 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:09.467825 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:09.468496 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:09.469840 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:09.470090 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:09.470213 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:09.471635 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:09.471643 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:09.471645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:09.471662 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:09.471685 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:09.471694 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:09.471787 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:09.471967 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 01:51:09.472088 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 01:51:09.473441 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:09.463751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37861 (local address 127.9.77.129:58971)
0810 01:51:09.464667 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:09.464717 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:09.466314 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:09.466917 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:09.466926 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:09.467028 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:09.467658 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:09.467673 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:09.468641 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:09.468650 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:09.469696 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:09.469707 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:09.470272 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:09.470285 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:09.470476 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:09.471306 (+ 830us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 01:51:09.471308 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:09.471359 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 01:51:09.471441 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:09.471782 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:09.471794 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:09.471804 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 01:51:09.471957 (+ 153us) client_negotiation.cc:240] Negotiation successful
0810 01:51:09.472346 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":182,"threads_started":1}
I20260810 01:51:09.475131 4710 heartbeater.cc:347] Connected to a master server at 127.9.77.190:37861
I20260810 01:51:09.475451 4710 heartbeater.cc:464] Registering TS with master...
I20260810 01:51:09.476173 4710 heartbeater.cc:511] Master 127.9.77.190:37861 requested a full tablet report, sending...
I20260810 01:51:09.478179 4416 ts_manager.cc:194] Registered new tserver with Master: 5e38ba31aafb454fab3a30af25642bd2 (127.9.77.129:33845)
I20260810 01:51:09.479614 4416 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.77.129:58971
W20260810 01:51:09.597612 4714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:09.597991 4714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:09.598065 4714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:09.602511 4714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:09.602607 4714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:09.602698 4714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:51:09.607681 4714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:34665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:37861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:09.609151 4714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:09.610937 4714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:09.623173 4721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:09.623273 4724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:09.623220 4714 server_base.cc:1034] running on GCE node
W20260810 01:51:09.623178 4722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:09.623989 4714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:09.624842 4714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:09.626055 4714 hybrid_clock.cc:648] HybridClock initialized: now 1786326669626029 us; error 36 us; skew 500 ppm
I20260810 01:51:09.628712 4714 webserver.cc:460] Webserver started at http://127.9.77.130:42981/ using document root <none> and password file <none>
I20260810 01:51:09.629578 4714 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:09.629684 4714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:09.630000 4714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:09.632578 4714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-1/data/instance:
uuid: "9d654d16d6ff4c47a9df9a62248a366a"
format_stamp: "Formatted at 2026-08-10 01:51:09 on dist-test-slave-kvfs"
server_key: "346dbc91f52214d6b0aef433a5eeecb8"
server_key_iv: "2d9b046b2fd58540339a954020a6b566"
server_key_version: "encryptionkey@0"
I20260810 01:51:09.633244 4714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance:
uuid: "9d654d16d6ff4c47a9df9a62248a366a"
format_stamp: "Formatted at 2026-08-10 01:51:09 on dist-test-slave-kvfs"
server_key: "346dbc91f52214d6b0aef433a5eeecb8"
server_key_iv: "2d9b046b2fd58540339a954020a6b566"
server_key_version: "encryptionkey@0"
I20260810 01:51:09.637756 4714 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 01:51:09.640782 4730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:09.642339 4714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:51:09.642517 4714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "9d654d16d6ff4c47a9df9a62248a366a"
format_stamp: "Formatted at 2026-08-10 01:51:09 on dist-test-slave-kvfs"
server_key: "346dbc91f52214d6b0aef433a5eeecb8"
server_key_iv: "2d9b046b2fd58540339a954020a6b566"
server_key_version: "encryptionkey@0"
I20260810 01:51:09.642673 4714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:09.658149 4714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:09.659096 4714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:09.659334 4714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:09.660141 4714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:51:09.661339 4714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:51:09.661417 4714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:09.661499 4714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:51:09.661551 4714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:09.672043 4714 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:34315
I20260810 01:51:09.672076 4890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:34315 every 8 connection(s)
I20260810 01:51:09.673141 4714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:51:09.674582 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 4714
I20260810 01:51:09.674716 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance
I20260810 01:51:09.675154 9526 external_mini_cluster.cc:1428] Setting key 1e4796bbdf083efc9a84de198fc4c692
I20260810 01:51:09.677814 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:0
--local_ip_for_outbound_sockets=127.9.77.131
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:37861
--builtin_ntp_servers=127.9.77.148:34665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 01:51:09.686184 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:09.675496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:49231 (local address 127.9.77.190:37861)
0810 01:51:09.675666 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:09.675670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:09.677238 (+ 1568us) server_negotiation.cc:407] Connection header received
0810 01:51:09.679022 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:09.679026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:09.679094 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:09.679215 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:09.680535 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:09.681137 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:09.683106 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:09.683446 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:09.683548 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:09.685357 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:09.685360 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:09.685371 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:09.685388 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:09.685457 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:09.685473 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:09.685560 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:09.685832 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 01:51:09.685944 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 01:51:09.688853 4894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:09.676101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37861 (local address 127.9.77.130:49231)
0810 01:51:09.677069 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:09.677135 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:09.678756 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:09.679431 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:09.679441 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:09.679546 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:09.680332 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:09.680358 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:09.681286 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:09.681292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:09.682949 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:09.682965 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:09.683634 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:09.683641 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:09.683905 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:09.684910 (+ 1005us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 01:51:09.684914 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:09.685002 (+ 88us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 01:51:09.685132 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:09.685619 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:09.685632 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:09.685639 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 01:51:09.686658 (+ 1019us) client_negotiation.cc:240] Negotiation successful
0810 01:51:09.687076 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":145,"threads_started":1}
I20260810 01:51:09.691319 4891 heartbeater.cc:347] Connected to a master server at 127.9.77.190:37861
I20260810 01:51:09.691670 4891 heartbeater.cc:464] Registering TS with master...
I20260810 01:51:09.692637 4891 heartbeater.cc:511] Master 127.9.77.190:37861 requested a full tablet report, sending...
I20260810 01:51:09.694286 4416 ts_manager.cc:194] Registered new tserver with Master: 9d654d16d6ff4c47a9df9a62248a366a (127.9.77.130:34315)
I20260810 01:51:09.695343 4416 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.77.130:49231
W20260810 01:51:09.839908 4895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:09.840389 4895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:09.840474 4895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:09.844885 4895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:09.844995 4895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:09.845101 4895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:51:09.850884 4895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:34665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:37861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:09.852643 4895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:09.854650 4895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:09.866940 4905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:09.867105 4895 server_base.cc:1034] running on GCE node
W20260810 01:51:09.866945 4903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:09.866940 4908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:09.867815 4895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:09.868674 4895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:09.869885 4895 hybrid_clock.cc:648] HybridClock initialized: now 1786326669869857 us; error 33 us; skew 500 ppm
I20260810 01:51:09.872346 4895 webserver.cc:460] Webserver started at http://127.9.77.131:42497/ using document root <none> and password file <none>
I20260810 01:51:09.873085 4895 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:09.873190 4895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:09.873525 4895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:09.876245 4895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-2/data/instance:
uuid: "08f9d90a2fd64eeda9d781c44e704d0c"
format_stamp: "Formatted at 2026-08-10 01:51:09 on dist-test-slave-kvfs"
server_key: "bfa769c78e306780907f435e70ea4a67"
server_key_iv: "9352467cbffd6ab9ec38aaddd347f245"
server_key_version: "encryptionkey@0"
I20260810 01:51:09.876910 4895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance:
uuid: "08f9d90a2fd64eeda9d781c44e704d0c"
format_stamp: "Formatted at 2026-08-10 01:51:09 on dist-test-slave-kvfs"
server_key: "bfa769c78e306780907f435e70ea4a67"
server_key_iv: "9352467cbffd6ab9ec38aaddd347f245"
server_key_version: "encryptionkey@0"
I20260810 01:51:09.881350 4895 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 01:51:09.884625 4916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:09.886158 4895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:51:09.886315 4895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "08f9d90a2fd64eeda9d781c44e704d0c"
format_stamp: "Formatted at 2026-08-10 01:51:09 on dist-test-slave-kvfs"
server_key: "bfa769c78e306780907f435e70ea4a67"
server_key_iv: "9352467cbffd6ab9ec38aaddd347f245"
server_key_version: "encryptionkey@0"
I20260810 01:51:09.886451 4895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:09.903872 4895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:09.904850 4895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:09.905112 4895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:09.905897 4895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:51:09.907131 4895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:51:09.907217 4895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:09.907305 4895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:51:09.907363 4895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:09.918381 4895 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:43269
I20260810 01:51:09.918414 5097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:43269 every 8 connection(s)
I20260810 01:51:09.919613 4895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:51:09.922382 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 4895
I20260810 01:51:09.922554 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance
I20260810 01:51:09.922924 9526 external_mini_cluster.cc:1428] Setting key 958d43eda41a4daaba5569745ac0604d
I20260810 01:51:09.931711 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:09.922162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:52615 (local address 127.9.77.190:37861)
0810 01:51:09.922357 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:09.922364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:09.923968 (+ 1604us) server_negotiation.cc:407] Connection header received
0810 01:51:09.925843 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:09.925846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:09.925903 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:09.926020 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:09.927266 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:09.927900 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:09.929487 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:09.929731 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:09.929826 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:09.931214 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:09.931217 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:09.931219 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:09.931233 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:09.931255 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:09.931264 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:09.931325 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:09.931485 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 01:51:09.931554 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 01:51:09.932963 5100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:09.922803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37861 (local address 127.9.77.131:52615)
0810 01:51:09.923777 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:09.923845 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:09.925604 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:09.926189 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:09.926196 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:09.926335 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:09.927089 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:09.927111 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:09.928094 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:09.928098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:09.929321 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:09.929342 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:09.929889 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:09.929893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:09.930082 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:09.930916 (+ 834us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 01:51:09.930918 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:09.930973 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 01:51:09.931058 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:09.931342 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:09.931351 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:09.931356 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 01:51:09.931474 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 01:51:09.931803 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":203,"threads_started":1}
I20260810 01:51:09.934310 5098 heartbeater.cc:347] Connected to a master server at 127.9.77.190:37861
I20260810 01:51:09.934618 5098 heartbeater.cc:464] Registering TS with master...
I20260810 01:51:09.935323 5098 heartbeater.cc:511] Master 127.9.77.190:37861 requested a full tablet report, sending...
I20260810 01:51:09.936698 4416 ts_manager.cc:194] Registered new tserver with Master: 08f9d90a2fd64eeda9d781c44e704d0c (127.9.77.131:43269)
I20260810 01:51:09.937446 4416 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.77.131:52615
I20260810 01:51:09.937495 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 01:51:09.944073 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:09.940153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33398 (local address 127.9.77.190:37861)
0810 01:51:09.940310 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:09.940316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:09.940483 (+ 167us) server_negotiation.cc:407] Connection header received
0810 01:51:09.940600 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:09.940602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:09.940648 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:09.940709 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:09.941028 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:09.941554 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:09.942430 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:09.942641 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:09.942716 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:09.943654 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:09.943656 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:09.943658 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:09.943670 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:09.943687 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:09.943695 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:09.943750 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:09.943802 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 01:51:09.943853 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 01:51:09.947600 4416 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 01:51:09.951607 4416 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:51:09.968914 5120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:09.963098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41868 (local address 127.9.77.131:43269)
0810 01:51:09.963726 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:09.963741 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:09.963765 (+ 24us) server_negotiation.cc:407] Connection header received
0810 01:51:09.963864 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:09.963869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:09.964120 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:09.964310 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:09.964671 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:09.965652 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:09.967386 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:09.968171 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:09.968306 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:09.968556 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:09.968613 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 01:51:09.968720 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":175,"threads_started":1}
I20260810 01:51:09.968906 5118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:09.962535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:43269 (local address 127.0.0.1:41868)
0810 01:51:09.963198 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:09.963252 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:09.963402 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:09.964301 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:09.964308 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:09.964332 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:09.964543 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:09.964550 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:09.965764 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:09.965771 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:09.967245 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:09.967255 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:09.968292 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:09.968296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:09.968405 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:09.968446 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 01:51:09.968554 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 01:51:09.968693 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":7,"thread_start_us":113,"threads_started":1}
I20260810 01:51:09.969329 5114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:09.962227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:34315 (local address 127.0.0.1:52658)
0810 01:51:09.963190 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:09.963253 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:09.963402 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:09.964346 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:09.964349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:09.964365 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:09.964551 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:09.964557 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:09.966056 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:09.966059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:09.967428 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:09.967442 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:09.968832 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:09.968836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:09.968997 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:09.969045 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 01:51:09.969119 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 01:51:09.969188 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"mutex_wait_us":117,"thread_start_us":97,"threads_started":1}
I20260810 01:51:09.969862 5119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:09.963098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:33845 (local address 127.0.0.1:59566)
0810 01:51:09.964223 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:09.964238 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:09.964347 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:09.964897 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:09.964900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:09.964920 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:09.965020 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:09.965026 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:09.966326 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:09.966329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:09.967556 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:09.967567 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:09.968832 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:09.968836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:09.969001 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:09.969581 (+ 580us) client_negotiation.cc:769] Sending connection context
0810 01:51:09.969654 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 01:51:09.969727 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1063,"mutex_wait_us":750,"thread_start_us":103,"threads_started":1}
I20260810 01:51:09.969928 5116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:09.962417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52658 (local address 127.9.77.130:34315)
0810 01:51:09.962899 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:09.962914 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:09.963534 (+ 620us) server_negotiation.cc:407] Connection header received
0810 01:51:09.963643 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:09.963648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:09.963922 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:09.964130 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:09.964681 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:09.965931 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:09.967559 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:09.968428 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:09.968562 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:09.968763 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:09.969625 (+ 862us) server_negotiation.cc:299] Negotiation successful
0810 01:51:09.969739 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":136,"threads_started":1}
I20260810 01:51:09.972517 5121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:09.963674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59566 (local address 127.9.77.129:33845)
0810 01:51:09.964049 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:09.964057 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:09.964433 (+ 376us) server_negotiation.cc:407] Connection header received
0810 01:51:09.964532 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:09.964537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:09.964730 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:09.964911 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:09.965124 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:09.966172 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:09.967691 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:09.968499 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:09.968596 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:09.968762 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:09.972199 (+ 3437us) server_negotiation.cc:299] Negotiation successful
0810 01:51:09.972329 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":122,"threads_started":1}
I20260810 01:51:09.972822 4989 tablet_service.cc:1467] Processing CreateTablet for tablet 04d9b600b93e42c5af7e8995db1ae1a6 (DEFAULT_TABLE table=test-table [id=e2a7a97306404a75bd2ac22db6b470d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:51:09.972837 4802 tablet_service.cc:1467] Processing CreateTablet for tablet 04d9b600b93e42c5af7e8995db1ae1a6 (DEFAULT_TABLE table=test-table [id=e2a7a97306404a75bd2ac22db6b470d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:51:09.974062 4802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d9b600b93e42c5af7e8995db1ae1a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:09.974294 4989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d9b600b93e42c5af7e8995db1ae1a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:09.975559 4609 tablet_service.cc:1467] Processing CreateTablet for tablet 04d9b600b93e42c5af7e8995db1ae1a6 (DEFAULT_TABLE table=test-table [id=e2a7a97306404a75bd2ac22db6b470d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:51:09.976725 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d9b600b93e42c5af7e8995db1ae1a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:09.981729 5123 tablet_bootstrap.cc:492] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a: Bootstrap starting.
I20260810 01:51:09.983301 5124 tablet_bootstrap.cc:492] T 04d9b600b93e42c5af7e8995db1ae1a6 P 08f9d90a2fd64eeda9d781c44e704d0c: Bootstrap starting.
I20260810 01:51:09.984673 5123 tablet_bootstrap.cc:654] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:09.984875 5125 tablet_bootstrap.cc:492] T 04d9b600b93e42c5af7e8995db1ae1a6 P 5e38ba31aafb454fab3a30af25642bd2: Bootstrap starting.
I20260810 01:51:09.985778 5124 tablet_bootstrap.cc:654] T 04d9b600b93e42c5af7e8995db1ae1a6 P 08f9d90a2fd64eeda9d781c44e704d0c: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:09.986541 5123 log.cc:824] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:09.986692 5124 log.cc:824] T 04d9b600b93e42c5af7e8995db1ae1a6 P 08f9d90a2fd64eeda9d781c44e704d0c: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:09.987426 5125 tablet_bootstrap.cc:654] T 04d9b600b93e42c5af7e8995db1ae1a6 P 5e38ba31aafb454fab3a30af25642bd2: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:09.988402 5125 log.cc:824] T 04d9b600b93e42c5af7e8995db1ae1a6 P 5e38ba31aafb454fab3a30af25642bd2: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:09.988972 5124 tablet_bootstrap.cc:492] T 04d9b600b93e42c5af7e8995db1ae1a6 P 08f9d90a2fd64eeda9d781c44e704d0c: No bootstrap required, opened a new log
I20260810 01:51:09.988996 5123 tablet_bootstrap.cc:492] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a: No bootstrap required, opened a new log
I20260810 01:51:09.989208 5124 ts_tablet_manager.cc:1397] T 04d9b600b93e42c5af7e8995db1ae1a6 P 08f9d90a2fd64eeda9d781c44e704d0c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 01:51:09.989216 5123 ts_tablet_manager.cc:1397] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260810 01:51:09.990577 5125 tablet_bootstrap.cc:492] T 04d9b600b93e42c5af7e8995db1ae1a6 P 5e38ba31aafb454fab3a30af25642bd2: No bootstrap required, opened a new log
I20260810 01:51:09.990773 5125 ts_tablet_manager.cc:1397] T 04d9b600b93e42c5af7e8995db1ae1a6 P 5e38ba31aafb454fab3a30af25642bd2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 01:51:09.993589 5123 raft_consensus.cc:348] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d654d16d6ff4c47a9df9a62248a366a" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 34315 } } peers { permanent_uuid: "08f9d90a2fd64eeda9d781c44e704d0c" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 43269 } } peers { permanent_uuid: "5e38ba31aafb454fab3a30af25642bd2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 33845 } }
I20260810 01:51:09.993950 5123 raft_consensus.cc:374] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:09.994017 5123 raft_consensus.cc:729] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d654d16d6ff4c47a9df9a62248a366a, State: Initialized, Role: FOLLOWER
I20260810 01:51:09.994220 5124 raft_consensus.cc:348] T 04d9b600b93e42c5af7e8995db1ae1a6 P 08f9d90a2fd64eeda9d781c44e704d0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d654d16d6ff4c47a9df9a62248a366a" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 34315 } } peers { permanent_uuid: "08f9d90a2fd64eeda9d781c44e704d0c" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 43269 } } peers { permanent_uuid: "5e38ba31aafb454fab3a30af25642bd2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 33845 } }
I20260810 01:51:09.994524 5124 raft_consensus.cc:374] T 04d9b600b93e42c5af7e8995db1ae1a6 P 08f9d90a2fd64eeda9d781c44e704d0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:09.994585 5124 raft_consensus.cc:729] T 04d9b600b93e42c5af7e8995db1ae1a6 P 08f9d90a2fd64eeda9d781c44e704d0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08f9d90a2fd64eeda9d781c44e704d0c, State: Initialized, Role: FOLLOWER
I20260810 01:51:09.994609 5123 consensus_queue.cc:260] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d654d16d6ff4c47a9df9a62248a366a" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 34315 } } peers { permanent_uuid: "08f9d90a2fd64eeda9d781c44e704d0c" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 43269 } } peers { permanent_uuid: "5e38ba31aafb454fab3a30af25642bd2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 33845 } }
I20260810 01:51:09.995085 5124 consensus_queue.cc:260] T 04d9b600b93e42c5af7e8995db1ae1a6 P 08f9d90a2fd64eeda9d781c44e704d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d654d16d6ff4c47a9df9a62248a366a" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 34315 } } peers { permanent_uuid: "08f9d90a2fd64eeda9d781c44e704d0c" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 43269 } } peers { permanent_uuid: "5e38ba31aafb454fab3a30af25642bd2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 33845 } }
I20260810 01:51:09.995247 5125 raft_consensus.cc:348] T 04d9b600b93e42c5af7e8995db1ae1a6 P 5e38ba31aafb454fab3a30af25642bd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d654d16d6ff4c47a9df9a62248a366a" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 34315 } } peers { permanent_uuid: "08f9d90a2fd64eeda9d781c44e704d0c" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 43269 } } peers { permanent_uuid: "5e38ba31aafb454fab3a30af25642bd2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 33845 } }
I20260810 01:51:09.995512 5125 raft_consensus.cc:374] T 04d9b600b93e42c5af7e8995db1ae1a6 P 5e38ba31aafb454fab3a30af25642bd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:09.995510 4891 heartbeater.cc:503] Master 127.9.77.190:37861 was elected leader, sending a full tablet report...
I20260810 01:51:09.995574 5125 raft_consensus.cc:729] T 04d9b600b93e42c5af7e8995db1ae1a6 P 5e38ba31aafb454fab3a30af25642bd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e38ba31aafb454fab3a30af25642bd2, State: Initialized, Role: FOLLOWER
I20260810 01:51:09.995967 5098 heartbeater.cc:503] Master 127.9.77.190:37861 was elected leader, sending a full tablet report...
I20260810 01:51:09.996261 5124 ts_tablet_manager.cc:1428] T 04d9b600b93e42c5af7e8995db1ae1a6 P 08f9d90a2fd64eeda9d781c44e704d0c: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 01:51:09.996740 5125 consensus_queue.cc:260] T 04d9b600b93e42c5af7e8995db1ae1a6 P 5e38ba31aafb454fab3a30af25642bd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d654d16d6ff4c47a9df9a62248a366a" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 34315 } } peers { permanent_uuid: "08f9d90a2fd64eeda9d781c44e704d0c" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 43269 } } peers { permanent_uuid: "5e38ba31aafb454fab3a30af25642bd2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 33845 } }
I20260810 01:51:09.997190 5123 ts_tablet_manager.cc:1428] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260810 01:51:09.997778 5125 ts_tablet_manager.cc:1428] T 04d9b600b93e42c5af7e8995db1ae1a6 P 5e38ba31aafb454fab3a30af25642bd2: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 01:51:09.998185 4710 heartbeater.cc:503] Master 127.9.77.190:37861 was elected leader, sending a full tablet report...
I20260810 01:51:10.065137 5129 raft_consensus.cc:482] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:51:10.065441 5129 raft_consensus.cc:504] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d654d16d6ff4c47a9df9a62248a366a" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 34315 } } peers { permanent_uuid: "08f9d90a2fd64eeda9d781c44e704d0c" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 43269 } } peers { permanent_uuid: "5e38ba31aafb454fab3a30af25642bd2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 33845 } }
I20260810 01:51:10.067010 5129 leader_election.cc:288] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08f9d90a2fd64eeda9d781c44e704d0c (127.9.77.131:43269), 5e38ba31aafb454fab3a30af25642bd2 (127.9.77.129:33845)
I20260810 01:51:10.072328 5120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:10.067509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:49905 (local address 127.9.77.131:43269)
0810 01:51:10.067658 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:10.067663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:10.067699 (+ 36us) server_negotiation.cc:407] Connection header received
0810 01:51:10.067872 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:10.067875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:10.067944 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:10.068094 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:10.068610 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:10.069419 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:10.070976 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:10.071643 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:10.071786 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:10.071856 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:10.072060 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 01:51:10.072161 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 01:51:10.072305 4894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:10.067364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:43269 (local address 127.9.77.130:49905)
0810 01:51:10.067551 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:10.067572 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:10.067713 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:10.068254 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:10.068258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:10.068282 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:10.068430 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:10.068456 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:10.069532 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:10.069534 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:10.070813 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:10.070825 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:10.071774 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:10.071776 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:10.071881 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:10.071920 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 01:51:10.072001 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 01:51:10.072116 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 01:51:10.072465 5135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:10.067749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:33845 (local address 127.9.77.130:45857)
0810 01:51:10.068422 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:10.068437 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:10.068557 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:10.068854 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:10.068856 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:10.068870 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:10.068992 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:10.068996 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:10.069863 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:10.069869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:10.071012 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:10.071020 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:10.071954 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:10.071960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:10.072100 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:10.072121 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 01:51:10.072197 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 01:51:10.072256 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":1,"thread_start_us":232,"threads_started":1}
I20260810 01:51:10.072458 5121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:10.068014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:45857 (local address 127.9.77.129:33845)
0810 01:51:10.068195 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:10.068201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:10.068569 (+ 368us) server_negotiation.cc:407] Connection header received
0810 01:51:10.068649 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:10.068652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:10.068714 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:10.068851 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:10.069110 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:10.069755 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:10.071140 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:10.071813 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:10.072039 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:10.072098 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:10.072226 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 01:51:10.072315 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 01:51:10.073357 5130 raft_consensus.cc:482] T 04d9b600b93e42c5af7e8995db1ae1a6 P 08f9d90a2fd64eeda9d781c44e704d0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:51:10.073331 5018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d9b600b93e42c5af7e8995db1ae1a6" candidate_uuid: "9d654d16d6ff4c47a9df9a62248a366a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08f9d90a2fd64eeda9d781c44e704d0c" is_pre_election: true
I20260810 01:51:10.073347 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d9b600b93e42c5af7e8995db1ae1a6" candidate_uuid: "9d654d16d6ff4c47a9df9a62248a366a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e38ba31aafb454fab3a30af25642bd2" is_pre_election: true
I20260810 01:51:10.073585 5130 raft_consensus.cc:504] T 04d9b600b93e42c5af7e8995db1ae1a6 P 08f9d90a2fd64eeda9d781c44e704d0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d654d16d6ff4c47a9df9a62248a366a" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 34315 } } peers { permanent_uuid: "08f9d90a2fd64eeda9d781c44e704d0c" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 43269 } } peers { permanent_uuid: "5e38ba31aafb454fab3a30af25642bd2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 33845 } }
I20260810 01:51:10.073716 4639 raft_consensus.cc:2454] T 04d9b600b93e42c5af7e8995db1ae1a6 P 5e38ba31aafb454fab3a30af25642bd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d654d16d6ff4c47a9df9a62248a366a in term 0.
I20260810 01:51:10.074141 5018 raft_consensus.cc:2454] T 04d9b600b93e42c5af7e8995db1ae1a6 P 08f9d90a2fd64eeda9d781c44e704d0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d654d16d6ff4c47a9df9a62248a366a in term 0.
I20260810 01:51:10.074357 4735 leader_election.cc:302] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e38ba31aafb454fab3a30af25642bd2, 9d654d16d6ff4c47a9df9a62248a366a; no voters:
I20260810 01:51:10.074762 5129 raft_consensus.cc:2781] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:51:10.074882 5129 raft_consensus.cc:482] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:51:10.074934 5129 raft_consensus.cc:3037] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:10.075270 5130 leader_election.cc:288] T 04d9b600b93e42c5af7e8995db1ae1a6 P 08f9d90a2fd64eeda9d781c44e704d0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d654d16d6ff4c47a9df9a62248a366a (127.9.77.130:34315), 5e38ba31aafb454fab3a30af25642bd2 (127.9.77.129:33845)
I20260810 01:51:10.076822 5129 raft_consensus.cc:504] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d654d16d6ff4c47a9df9a62248a366a" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 34315 } } peers { permanent_uuid: "08f9d90a2fd64eeda9d781c44e704d0c" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 43269 } } peers { permanent_uuid: "5e38ba31aafb454fab3a30af25642bd2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 33845 } }
I20260810 01:51:10.077311 5129 leader_election.cc:288] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a [CANDIDATE]: Term 1 election: Requested vote from peers 08f9d90a2fd64eeda9d781c44e704d0c (127.9.77.131:43269), 5e38ba31aafb454fab3a30af25642bd2 (127.9.77.129:33845)
I20260810 01:51:10.077747 5018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d9b600b93e42c5af7e8995db1ae1a6" candidate_uuid: "9d654d16d6ff4c47a9df9a62248a366a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08f9d90a2fd64eeda9d781c44e704d0c"
I20260810 01:51:10.077883 5018 raft_consensus.cc:3037] T 04d9b600b93e42c5af7e8995db1ae1a6 P 08f9d90a2fd64eeda9d781c44e704d0c [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:10.078017 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d9b600b93e42c5af7e8995db1ae1a6" candidate_uuid: "9d654d16d6ff4c47a9df9a62248a366a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e38ba31aafb454fab3a30af25642bd2"
I20260810 01:51:10.078172 4639 raft_consensus.cc:3037] T 04d9b600b93e42c5af7e8995db1ae1a6 P 5e38ba31aafb454fab3a30af25642bd2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:10.079392 5018 raft_consensus.cc:2454] T 04d9b600b93e42c5af7e8995db1ae1a6 P 08f9d90a2fd64eeda9d781c44e704d0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d654d16d6ff4c47a9df9a62248a366a in term 1.
I20260810 01:51:10.079689 4639 raft_consensus.cc:2454] T 04d9b600b93e42c5af7e8995db1ae1a6 P 5e38ba31aafb454fab3a30af25642bd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d654d16d6ff4c47a9df9a62248a366a in term 1.
I20260810 01:51:10.079746 4735 leader_election.cc:302] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08f9d90a2fd64eeda9d781c44e704d0c, 9d654d16d6ff4c47a9df9a62248a366a; no voters:
I20260810 01:51:10.080070 5129 raft_consensus.cc:2781] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:10.080407 5129 raft_consensus.cc:686] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a [term 1 LEADER]: Becoming Leader. State: Replica: 9d654d16d6ff4c47a9df9a62248a366a, State: Running, Role: LEADER
I20260810 01:51:10.080667 5116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:10.075719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:39253 (local address 127.9.77.130:34315)
0810 01:51:10.075831 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:10.075835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:10.076494 (+ 659us) server_negotiation.cc:407] Connection header received
0810 01:51:10.076663 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:10.076666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:10.076712 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:10.076755 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:10.077155 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:10.077855 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:10.079307 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:10.079947 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:10.080067 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:10.080134 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:10.080500 (+ 366us) server_negotiation.cc:299] Negotiation successful
0810 01:51:10.080571 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 01:51:10.080665 5121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:10.075609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:45613 (local address 127.9.77.129:33845)
0810 01:51:10.075749 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:10.075755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:10.075770 (+ 15us) server_negotiation.cc:407] Connection header received
0810 01:51:10.075815 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:10.075818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:10.075869 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:10.075965 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:10.076290 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:10.077039 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:10.079111 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:10.079754 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:10.079873 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:10.079941 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:10.080450 (+ 509us) server_negotiation.cc:299] Negotiation successful
0810 01:51:10.080533 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 01:51:10.080669 5100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:10.075465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:33845 (local address 127.9.77.131:45613)
0810 01:51:10.075600 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:10.075613 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:10.075714 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:10.075982 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:10.075986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:10.076006 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:10.076149 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:10.076158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:10.077747 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:10.077751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:10.078953 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:10.078967 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:10.080128 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:10.080135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:10.080272 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:10.080310 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 01:51:10.080418 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 01:51:10.080521 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 01:51:10.080905 5129 consensus_queue.cc:237] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d654d16d6ff4c47a9df9a62248a366a" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 34315 } } peers { permanent_uuid: "08f9d90a2fd64eeda9d781c44e704d0c" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 43269 } } peers { permanent_uuid: "5e38ba31aafb454fab3a30af25642bd2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 33845 } }
I20260810 01:51:10.081066 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:10.075465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:34315 (local address 127.9.77.131:39253)
0810 01:51:10.076399 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:10.076414 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:10.076557 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:10.076873 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:10.076876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:10.076890 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:10.077027 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:10.077032 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:10.078009 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:10.078012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:10.079155 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:10.079167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:10.080233 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:10.080237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:10.080351 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:10.080374 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 01:51:10.080788 (+ 414us) client_negotiation.cc:240] Negotiation successful
0810 01:51:10.080881 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":320,"thread_start_us":162,"threads_started":1}
I20260810 01:51:10.081657 4639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d9b600b93e42c5af7e8995db1ae1a6" candidate_uuid: "08f9d90a2fd64eeda9d781c44e704d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e38ba31aafb454fab3a30af25642bd2" is_pre_election: true
I20260810 01:51:10.081861 4639 raft_consensus.cc:2379] T 04d9b600b93e42c5af7e8995db1ae1a6 P 5e38ba31aafb454fab3a30af25642bd2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 08f9d90a2fd64eeda9d781c44e704d0c in current term 1: Already voted for candidate 9d654d16d6ff4c47a9df9a62248a366a in this term.
I20260810 01:51:10.082649 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d9b600b93e42c5af7e8995db1ae1a6" candidate_uuid: "08f9d90a2fd64eeda9d781c44e704d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d654d16d6ff4c47a9df9a62248a366a" is_pre_election: true
I20260810 01:51:10.083264 4921 leader_election.cc:302] T 04d9b600b93e42c5af7e8995db1ae1a6 P 08f9d90a2fd64eeda9d781c44e704d0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08f9d90a2fd64eeda9d781c44e704d0c; no voters: 5e38ba31aafb454fab3a30af25642bd2, 9d654d16d6ff4c47a9df9a62248a366a
I20260810 01:51:10.083570 5130 raft_consensus.cc:2726] T 04d9b600b93e42c5af7e8995db1ae1a6 P 08f9d90a2fd64eeda9d781c44e704d0c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 01:51:10.085721 4413 catalog_manager.cc:5495] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d654d16d6ff4c47a9df9a62248a366a (127.9.77.130). New cstate: current_term: 1 leader_uuid: "9d654d16d6ff4c47a9df9a62248a366a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d654d16d6ff4c47a9df9a62248a366a" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 34315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08f9d90a2fd64eeda9d781c44e704d0c" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 43269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e38ba31aafb454fab3a30af25642bd2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 33845 } health_report { overall_health: UNKNOWN } } }
I20260810 01:51:10.126955 5116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:10.122486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52660 (local address 127.9.77.130:34315)
0810 01:51:10.122896 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:10.122901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:10.122918 (+ 17us) server_negotiation.cc:407] Connection header received
0810 01:51:10.123023 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:10.123026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:10.123100 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:10.123206 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:51:10.123516 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:10.124094 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:10.125119 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:10.125370 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:10.125467 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:10.125820 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:51:10.126418 (+ 598us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:51:10.126584 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:10.126628 (+ 44us) server_negotiation.cc:299] Negotiation successful
0810 01:51:10.126694 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285}
I20260810 01:51:10.135219 4639 raft_consensus.cc:1261] T 04d9b600b93e42c5af7e8995db1ae1a6 P 5e38ba31aafb454fab3a30af25642bd2 [term 1 FOLLOWER]: Refusing update from remote peer 9d654d16d6ff4c47a9df9a62248a366a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:51:10.135273 5018 raft_consensus.cc:1261] T 04d9b600b93e42c5af7e8995db1ae1a6 P 08f9d90a2fd64eeda9d781c44e704d0c [term 1 FOLLOWER]: Refusing update from remote peer 9d654d16d6ff4c47a9df9a62248a366a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:51:10.136128 5138 consensus_queue.cc:1035] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e38ba31aafb454fab3a30af25642bd2" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 33845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:51:10.136416 5129 consensus_queue.cc:1035] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a [LEADER]: Connected to new peer: Peer: permanent_uuid: "08f9d90a2fd64eeda9d781c44e704d0c" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 43269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:51:10.146476 5146 mvcc.cc:205] Tried to move back new op lower bound from 7316794040864350208 to 7316794040655380480. Current Snapshot: MvccSnapshot[applied={T|T < 7316794040864350208}]
I20260810 01:51:10.149197 5144 mvcc.cc:205] Tried to move back new op lower bound from 7316794040864350208 to 7316794040655380480. Current Snapshot: MvccSnapshot[applied={T|T < 7316794040864350208}]
I20260810 01:51:10.156908 4414 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:51:10.157167 4414 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:51:10.160532 4414 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fcd5fa2e772d46cab4e8d57322d1feb0: Sending DeleteTablet for 3 replicas of tablet 04d9b600b93e42c5af7e8995db1ae1a6
I20260810 01:51:10.161651 4802 tablet_service.cc:1514] Processing DeleteTablet for tablet 04d9b600b93e42c5af7e8995db1ae1a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:51:10 UTC) from {username='slave'} at 127.0.0.1:52658
I20260810 01:51:10.161813 4989 tablet_service.cc:1514] Processing DeleteTablet for tablet 04d9b600b93e42c5af7e8995db1ae1a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:51:10 UTC) from {username='slave'} at 127.0.0.1:41868
I20260810 01:51:10.162382 4609 tablet_service.cc:1514] Processing DeleteTablet for tablet 04d9b600b93e42c5af7e8995db1ae1a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:51:10 UTC) from {username='slave'} at 127.0.0.1:59566
I20260810 01:51:10.162791 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 4503
I20260810 01:51:10.163025 5161 tablet_replica.cc:330] T 04d9b600b93e42c5af7e8995db1ae1a6 P 5e38ba31aafb454fab3a30af25642bd2: stopping tablet replica
I20260810 01:51:10.177098 5159 tablet_replica.cc:330] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a: stopping tablet replica
I20260810 01:51:10.177590 5159 raft_consensus.cc:2229] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a [term 1 LEADER]: Raft consensus shutting down.
I20260810 01:51:10.178203 5159 raft_consensus.cc:2258] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:51:10.179250 5160 tablet_replica.cc:330] T 04d9b600b93e42c5af7e8995db1ae1a6 P 08f9d90a2fd64eeda9d781c44e704d0c: stopping tablet replica
I20260810 01:51:10.179654 5160 raft_consensus.cc:2229] T 04d9b600b93e42c5af7e8995db1ae1a6 P 08f9d90a2fd64eeda9d781c44e704d0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:51:10.180091 5160 raft_consensus.cc:2258] T 04d9b600b93e42c5af7e8995db1ae1a6 P 08f9d90a2fd64eeda9d781c44e704d0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:51:10.180536 5159 ts_tablet_manager.cc:1905] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:51:10.181728 5160 ts_tablet_manager.cc:1905] T 04d9b600b93e42c5af7e8995db1ae1a6 P 08f9d90a2fd64eeda9d781c44e704d0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:51:10.185003 5160 ts_tablet_manager.cc:1918] T 04d9b600b93e42c5af7e8995db1ae1a6 P 08f9d90a2fd64eeda9d781c44e704d0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:51:10.185118 5160 log.cc:1196] T 04d9b600b93e42c5af7e8995db1ae1a6 P 08f9d90a2fd64eeda9d781c44e704d0c: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-2/wal/wals/04d9b600b93e42c5af7e8995db1ae1a6
I20260810 01:51:10.185019 5159 ts_tablet_manager.cc:1918] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:51:10.185350 5159 log.cc:1196] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786326392877349-9526-0/minicluster-data/ts-1/wal/wals/04d9b600b93e42c5af7e8995db1ae1a6
I20260810 01:51:10.185480 5160 ts_tablet_manager.cc:1939] T 04d9b600b93e42c5af7e8995db1ae1a6 P 08f9d90a2fd64eeda9d781c44e704d0c: Deleting consensus metadata
I20260810 01:51:10.185832 5159 ts_tablet_manager.cc:1939] T 04d9b600b93e42c5af7e8995db1ae1a6 P 9d654d16d6ff4c47a9df9a62248a366a: Deleting consensus metadata
I20260810 01:51:10.187146 4394 catalog_manager.cc:4841] TS 08f9d90a2fd64eeda9d781c44e704d0c (127.9.77.131:43269): tablet 04d9b600b93e42c5af7e8995db1ae1a6 (table test-table [id=e2a7a97306404a75bd2ac22db6b470d0]) successfully deleted
I20260810 01:51:10.187146 4391 catalog_manager.cc:4841] TS 9d654d16d6ff4c47a9df9a62248a366a (127.9.77.130:34315): tablet 04d9b600b93e42c5af7e8995db1ae1a6 (table test-table [id=e2a7a97306404a75bd2ac22db6b470d0]) successfully deleted
W20260810 01:51:10.187584 4394 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.77.129:33845 (error 108)
I20260810 01:51:10.187913 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 4714
I20260810 01:51:10.199787 5119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:10.199111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:33845 (local address 127.0.0.1:59576)
0810 01:51:10.199515 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:10.199631 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.129:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245}
W20260810 01:51:10.200309 4394 catalog_manager.cc:4584] TS 5e38ba31aafb454fab3a30af25642bd2 (127.9.77.129:33845): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 04d9b600b93e42c5af7e8995db1ae1a6: Network error: Client connection negotiation failed: client connection to 127.9.77.129:33845: connect: Connection refused (error 111)
I20260810 01:51:10.218372 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 4895
I20260810 01:51:10.243492 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 4370
2026-08-10T01:51:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1407 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:51:10 dist-test-slave-kvfs krb5kdc[5168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:51:10 dist-test-slave-kvfs krb5kdc[5168](info): set up 2 sockets
Aug 10 01:51:10 dist-test-slave-kvfs krb5kdc[5168](info): commencing operation
krb5kdc: starting...
W20260810 01:51:11.543576 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.261s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:51:11 dist-test-slave-kvfs krb5kdc[5168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:51:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:51:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:51:11.695832 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:39299
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:39299
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:11.830245 5188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:11.830665 5188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:11.830746 5188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:11.835242 5188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:11.835332 5188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:11.835361 5188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:11.835386 5188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:11.835408 5188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:11.841372 5188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:39299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:39299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:11.842734 5188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:11.844079 5188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:11.854645 5200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:11.854645 5196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:11.854743 5188 server_base.cc:1034] running on GCE node
W20260810 01:51:11.854645 5202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:11.855681 5188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:11.857220 5188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:11.858484 5188 hybrid_clock.cc:648] HybridClock initialized: now 1786326671858449 us; error 49 us; skew 500 ppm
Aug 10 01:51:11 dist-test-slave-kvfs krb5kdc[5168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326671, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:11.868393 5188 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:51:11.869748 5188 webserver.cc:460] Webserver started at http://127.9.77.190:44211/ using document root <none> and password file <none>
I20260810 01:51:11.870546 5188 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:11.870621 5188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:11.870934 5188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:11.874367 5188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "e879a69b4c8e44a1a6ae1622dadb405a"
format_stamp: "Formatted at 2026-08-10 01:51:11 on dist-test-slave-kvfs"
server_key: "67fb3b2c5653a0041747c6bcce7566c9"
server_key_iv: "6cdd9abed026379bce31b6b63960d386"
server_key_version: "encryptionkey@0"
I20260810 01:51:11.875098 5188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "e879a69b4c8e44a1a6ae1622dadb405a"
format_stamp: "Formatted at 2026-08-10 01:51:11 on dist-test-slave-kvfs"
server_key: "67fb3b2c5653a0041747c6bcce7566c9"
server_key_iv: "6cdd9abed026379bce31b6b63960d386"
server_key_version: "encryptionkey@0"
I20260810 01:51:11.879812 5188 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 01:51:11.882879 5216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:11.884426 5188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:51:11.884585 5188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "e879a69b4c8e44a1a6ae1622dadb405a"
format_stamp: "Formatted at 2026-08-10 01:51:11 on dist-test-slave-kvfs"
server_key: "67fb3b2c5653a0041747c6bcce7566c9"
server_key_iv: "6cdd9abed026379bce31b6b63960d386"
server_key_version: "encryptionkey@0"
I20260810 01:51:11.884720 5188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:11.902899 5188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:11.907048 5188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:11.907305 5188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:11.916590 5188 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:39299
I20260810 01:51:11.916651 5297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:39299 every 8 connection(s)
I20260810 01:51:11.917834 5188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:51:11.921547 5300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:11.925993 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 5188
I20260810 01:51:11.926188 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:51:11.926568 9526 external_mini_cluster.cc:1428] Setting key 4dd111067c798a2e3d6dec96e45f4ce3
I20260810 01:51:11.929615 5300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e879a69b4c8e44a1a6ae1622dadb405a: Bootstrap starting.
I20260810 01:51:11.932878 5300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e879a69b4c8e44a1a6ae1622dadb405a: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:11.933869 5300 log.cc:824] T 00000000000000000000000000000000 P e879a69b4c8e44a1a6ae1622dadb405a: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:11.936795 5300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e879a69b4c8e44a1a6ae1622dadb405a: No bootstrap required, opened a new log
Aug 10 01:51:11 dist-test-slave-kvfs krb5kdc[5168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:51:11.941753 5300 raft_consensus.cc:348] T 00000000000000000000000000000000 P e879a69b4c8e44a1a6ae1622dadb405a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e879a69b4c8e44a1a6ae1622dadb405a" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 39299 } }
I20260810 01:51:11.942142 5300 raft_consensus.cc:374] T 00000000000000000000000000000000 P e879a69b4c8e44a1a6ae1622dadb405a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:11.942245 5300 raft_consensus.cc:729] T 00000000000000000000000000000000 P e879a69b4c8e44a1a6ae1622dadb405a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e879a69b4c8e44a1a6ae1622dadb405a, State: Initialized, Role: FOLLOWER
I20260810 01:51:11.942885 5300 consensus_queue.cc:260] T 00000000000000000000000000000000 P e879a69b4c8e44a1a6ae1622dadb405a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e879a69b4c8e44a1a6ae1622dadb405a" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 39299 } }
I20260810 01:51:11.943091 5300 raft_consensus.cc:388] T 00000000000000000000000000000000 P e879a69b4c8e44a1a6ae1622dadb405a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:51:11.943198 5300 raft_consensus.cc:482] T 00000000000000000000000000000000 P e879a69b4c8e44a1a6ae1622dadb405a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:51:11.943352 5300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e879a69b4c8e44a1a6ae1622dadb405a [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:11.944777 5300 raft_consensus.cc:504] T 00000000000000000000000000000000 P e879a69b4c8e44a1a6ae1622dadb405a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e879a69b4c8e44a1a6ae1622dadb405a" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 39299 } }
I20260810 01:51:11.945127 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:11.928452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40752 (local address 127.9.77.190:39299)
0810 01:51:11.928958 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:11.928977 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:11.929022 (+ 45us) server_negotiation.cc:407] Connection header received
0810 01:51:11.929891 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:11.929914 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:11.930318 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:11.930653 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:11.931708 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:11.932960 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:11.933925 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:11.934589 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:11.934770 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:11.938128 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:11.938135 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:11.938156 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:11.938194 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:11.941257 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:11.941738 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:11.941745 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:11.941758 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:11.941918 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:11.942226 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:11.942230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:11.942231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:11.942884 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:11.943129 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:11.943460 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 01:51:11.943854 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":151,"threads_started":1}
I20260810 01:51:11.945214 5300 leader_election.cc:302] T 00000000000000000000000000000000 P e879a69b4c8e44a1a6ae1622dadb405a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e879a69b4c8e44a1a6ae1622dadb405a; no voters:
I20260810 01:51:11.945683 5300 leader_election.cc:288] T 00000000000000000000000000000000 P e879a69b4c8e44a1a6ae1622dadb405a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:51:11.945815 5305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e879a69b4c8e44a1a6ae1622dadb405a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:11.946079 5305 raft_consensus.cc:686] T 00000000000000000000000000000000 P e879a69b4c8e44a1a6ae1622dadb405a [term 1 LEADER]: Becoming Leader. State: Replica: e879a69b4c8e44a1a6ae1622dadb405a, State: Running, Role: LEADER
I20260810 01:51:11.946520 5305 consensus_queue.cc:237] T 00000000000000000000000000000000 P e879a69b4c8e44a1a6ae1622dadb405a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e879a69b4c8e44a1a6ae1622dadb405a" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 39299 } }
I20260810 01:51:11.947279 5300 sys_catalog.cc:564] T 00000000000000000000000000000000 P e879a69b4c8e44a1a6ae1622dadb405a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:11.948987 5306 sys_catalog.cc:455] T 00000000000000000000000000000000 P e879a69b4c8e44a1a6ae1622dadb405a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e879a69b4c8e44a1a6ae1622dadb405a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e879a69b4c8e44a1a6ae1622dadb405a" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 39299 } } }
I20260810 01:51:11.949024 5307 sys_catalog.cc:455] T 00000000000000000000000000000000 P e879a69b4c8e44a1a6ae1622dadb405a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e879a69b4c8e44a1a6ae1622dadb405a. Latest consensus state: current_term: 1 leader_uuid: "e879a69b4c8e44a1a6ae1622dadb405a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e879a69b4c8e44a1a6ae1622dadb405a" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 39299 } } }
I20260810 01:51:11.949119 5306 sys_catalog.cc:458] T 00000000000000000000000000000000 P e879a69b4c8e44a1a6ae1622dadb405a [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:11.949132 5307 sys_catalog.cc:458] T 00000000000000000000000000000000 P e879a69b4c8e44a1a6ae1622dadb405a [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:11.949577 5318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:51:11.953106 5318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:51:11.959836 5318 catalog_manager.cc:1347] Generated new cluster ID: c307b48aa74d47f09fc15714164a2fdd
I20260810 01:51:11.959929 5318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:51:11.985702 5318 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:51:11.986922 5318 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:51:11.992264 5318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e879a69b4c8e44a1a6ae1622dadb405a: Generated new TSK 0
I20260810 01:51:11.993294 5318 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.77.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
I20260810 01:51:12.046921 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:0
--local_ip_for_outbound_sockets=127.9.77.129
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:39299
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:12.190243 5334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:12.190572 5334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:12.190615 5334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:12.194712 5334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:12.194774 5334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:12.194877 5334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:51:12.200343 5334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:39299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:12.201678 5334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:12.202989 5334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 01:51:12.214560 5334 server_base.cc:1034] running on GCE node
W20260810 01:51:12.214578 5342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:12.214733 5341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:12.214578 5345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:12.215242 5334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:12.215937 5334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:12.217149 5334 hybrid_clock.cc:648] HybridClock initialized: now 1786326672217125 us; error 32 us; skew 500 ppm
Aug 10 01:51:12 dist-test-slave-kvfs krb5kdc[5168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326672, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:12.220451 5334 init.cc:377] Logged in from keytab as kudu/127.9.77.129@KRBTEST.COM (short username kudu)
I20260810 01:51:12.221529 5334 webserver.cc:460] Webserver started at http://127.9.77.129:32791/ using document root <none> and password file <none>
I20260810 01:51:12.222245 5334 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:12.222313 5334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:12.222597 5334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:12.225232 5334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-0/data/instance:
uuid: "93e19962e32046cab363a72ec831cbc7"
format_stamp: "Formatted at 2026-08-10 01:51:12 on dist-test-slave-kvfs"
server_key: "fb6693010c4fb5ed41e8be3c57f5260e"
server_key_iv: "be71e0c60b0ba9b908b7cb93cc998fcf"
server_key_version: "encryptionkey@0"
I20260810 01:51:12.225919 5334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance:
uuid: "93e19962e32046cab363a72ec831cbc7"
format_stamp: "Formatted at 2026-08-10 01:51:12 on dist-test-slave-kvfs"
server_key: "fb6693010c4fb5ed41e8be3c57f5260e"
server_key_iv: "be71e0c60b0ba9b908b7cb93cc998fcf"
server_key_version: "encryptionkey@0"
I20260810 01:51:12.230674 5334 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 01:51:12.233680 5353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:12.235288 5334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 01:51:12.235442 5334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "93e19962e32046cab363a72ec831cbc7"
format_stamp: "Formatted at 2026-08-10 01:51:12 on dist-test-slave-kvfs"
server_key: "fb6693010c4fb5ed41e8be3c57f5260e"
server_key_iv: "be71e0c60b0ba9b908b7cb93cc998fcf"
server_key_version: "encryptionkey@0"
I20260810 01:51:12.235582 5334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:12.247648 5334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:12.251312 5334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:12.251525 5334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:12.252288 5334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:51:12.253291 5334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:51:12.253343 5334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:12.253414 5334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:51:12.253445 5334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:12.264551 5334 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:37217
I20260810 01:51:12.264626 5505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:37217 every 8 connection(s)
I20260810 01:51:12.265718 5334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:51:12.267933 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 5334
I20260810 01:51:12.268180 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance
I20260810 01:51:12.268671 9526 external_mini_cluster.cc:1428] Setting key d14cb92b26659fc76bc294167ddf0c24
Aug 10 01:51:12 dist-test-slave-kvfs krb5kdc[5168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326672, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.77.130@KRBTEST.COM; defaulting to no policy
I20260810 01:51:12.283871 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:12.268584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:44077 (local address 127.9.77.190:39299)
0810 01:51:12.268801 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:12.268809 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:12.269992 (+ 1183us) server_negotiation.cc:407] Connection header received
0810 01:51:12.272219 (+ 2227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:12.272221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:12.272285 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:12.272352 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:12.274188 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:12.274794 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:12.276087 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:12.276356 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:12.276477 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:12.279483 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:12.279487 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:12.279489 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:12.279517 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:12.281284 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:12.281877 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:12.281881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:12.281884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:12.281979 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:12.282354 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:12.282358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:12.282360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:12.282585 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:12.282703 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:12.283378 (+ 675us) server_negotiation.cc:299] Negotiation successful
0810 01:51:12.283622 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 01:51:12.285108 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:12.269134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:39299 (local address 127.9.77.129:44077)
0810 01:51:12.269827 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:12.269880 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:12.271914 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:12.272575 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:12.272587 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:12.273203 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:12.274024 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:12.274043 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:12.274941 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:12.274944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:12.275924 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:12.275934 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:12.276528 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:12.276533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:12.276733 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:12.277454 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:51:12.277473 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:12.279260 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:12.281423 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:12.281430 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:12.281450 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:12.281744 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:12.282078 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:12.282082 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:12.282084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:12.282247 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:12.282700 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:12.282708 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:12.283027 (+ 319us) client_negotiation.cc:769] Sending connection context
0810 01:51:12.283317 (+ 290us) client_negotiation.cc:240] Negotiation successful
0810 01:51:12.283763 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":184,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.130@KRBTEST.COM" created.
I20260810 01:51:12.287016 5507 heartbeater.cc:347] Connected to a master server at 127.9.77.190:39299
I20260810 01:51:12.287544 5507 heartbeater.cc:464] Registering TS with master...
I20260810 01:51:12.288369 5507 heartbeater.cc:511] Master 127.9.77.190:39299 requested a full tablet report, sending...
I20260810 01:51:12.290534 5244 ts_manager.cc:194] Registered new tserver with Master: 93e19962e32046cab363a72ec831cbc7 (127.9.77.129:37217)
I20260810 01:51:12.291956 5244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.129@KRBTEST.COM'} at 127.9.77.129:44077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
I20260810 01:51:12.307828 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:0
--local_ip_for_outbound_sockets=127.9.77.130
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:39299
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:12.444270 5516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:12.444628 5516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:12.444703 5516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:12.449400 5516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:12.449512 5516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:12.449617 5516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:51:12.455653 5516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:39299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:12.457221 5516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:12.458562 5516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:12.470772 5525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:12.470775 5526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:12.470772 5530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:12.471268 5516 server_base.cc:1034] running on GCE node
I20260810 01:51:12.471791 5516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:12.472653 5516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:12.473834 5516 hybrid_clock.cc:648] HybridClock initialized: now 1786326672473807 us; error 39 us; skew 500 ppm
Aug 10 01:51:12 dist-test-slave-kvfs krb5kdc[5168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326672, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:12.477437 5516 init.cc:377] Logged in from keytab as kudu/127.9.77.130@KRBTEST.COM (short username kudu)
I20260810 01:51:12.478583 5516 webserver.cc:460] Webserver started at http://127.9.77.130:34287/ using document root <none> and password file <none>
I20260810 01:51:12.479306 5516 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:12.479377 5516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:12.479643 5516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:12.482297 5516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-1/data/instance:
uuid: "77c2879ddf82486aafb429134defbdbe"
format_stamp: "Formatted at 2026-08-10 01:51:12 on dist-test-slave-kvfs"
server_key: "88359888b9c30382e1c874acfaee22dc"
server_key_iv: "70360716a93d7a65c28ea6d9f01fe194"
server_key_version: "encryptionkey@0"
I20260810 01:51:12.483007 5516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance:
uuid: "77c2879ddf82486aafb429134defbdbe"
format_stamp: "Formatted at 2026-08-10 01:51:12 on dist-test-slave-kvfs"
server_key: "88359888b9c30382e1c874acfaee22dc"
server_key_iv: "70360716a93d7a65c28ea6d9f01fe194"
server_key_version: "encryptionkey@0"
I20260810 01:51:12.487617 5516 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 01:51:12.490721 5542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:12.492383 5516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:51:12.492527 5516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "77c2879ddf82486aafb429134defbdbe"
format_stamp: "Formatted at 2026-08-10 01:51:12 on dist-test-slave-kvfs"
server_key: "88359888b9c30382e1c874acfaee22dc"
server_key_iv: "70360716a93d7a65c28ea6d9f01fe194"
server_key_version: "encryptionkey@0"
I20260810 01:51:12.492658 5516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:12.512311 5516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:12.515980 5516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:12.516251 5516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:12.517086 5516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:51:12.518203 5516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:51:12.518277 5516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:12.518358 5516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:51:12.518411 5516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:12.529255 5516 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:44989
I20260810 01:51:12.529286 5706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:44989 every 8 connection(s)
I20260810 01:51:12.530403 5516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:51:12.538388 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 5516
I20260810 01:51:12.538584 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance
I20260810 01:51:12.539016 9526 external_mini_cluster.cc:1428] Setting key a21fb2a293e929a8cbe25e86d0c408f6
Aug 10 01:51:12 dist-test-slave-kvfs krb5kdc[5168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326672, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:51:12.552372 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:12.532911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:46035 (local address 127.9.77.190:39299)
0810 01:51:12.533100 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:12.533105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:12.534191 (+ 1086us) server_negotiation.cc:407] Connection header received
0810 01:51:12.536549 (+ 2358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:12.536554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:12.536680 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:12.536794 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:12.539010 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:12.540088 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:12.542347 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:12.542784 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:12.542937 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:12.546894 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:12.546899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:12.546902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:12.546958 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:12.549228 (+ 2270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:12.549965 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:12.549969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:12.549971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:12.550088 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:12.550631 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:12.550635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:12.550637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:12.551035 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:12.551172 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:12.551997 (+ 825us) server_negotiation.cc:299] Negotiation successful
0810 01:51:12.552199 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 01:51:12.553984 5709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:12.533424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:39299 (local address 127.9.77.130:46035)
0810 01:51:12.534003 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:12.534055 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:12.536220 (+ 2165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:12.537094 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:12.537107 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:12.537795 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:12.538759 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:12.538786 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:12.540375 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:12.540381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:12.542134 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:12.542150 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:12.543020 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:12.543025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:12.543305 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:12.544235 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:51:12.544271 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:12.546628 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:12.549370 (+ 2742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:12.549381 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:12.549409 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:12.549833 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:12.550223 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:12.550229 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:12.550234 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:12.550471 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:12.551192 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:12.551202 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:12.551665 (+ 463us) client_negotiation.cc:769] Sending connection context
0810 01:51:12.551965 (+ 300us) client_negotiation.cc:240] Negotiation successful
0810 01:51:12.552478 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":145,"threads_started":1}
WARNING: no policy specified for kudu/127.9.77.131@KRBTEST.COM; defaulting to no policy
I20260810 01:51:12.555727 5707 heartbeater.cc:347] Connected to a master server at 127.9.77.190:39299
I20260810 01:51:12.556012 5707 heartbeater.cc:464] Registering TS with master...
I20260810 01:51:12.556768 5707 heartbeater.cc:511] Master 127.9.77.190:39299 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.131@KRBTEST.COM" created.
I20260810 01:51:12.558284 5244 ts_manager.cc:194] Registered new tserver with Master: 77c2879ddf82486aafb429134defbdbe (127.9.77.130:44989)
I20260810 01:51:12.559022 5244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.130@KRBTEST.COM'} at 127.9.77.130:46035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
I20260810 01:51:12.575829 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:0
--local_ip_for_outbound_sockets=127.9.77.131
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:39299
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:12.711086 5716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:12.711462 5716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:12.711509 5716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:12.715736 5716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:12.715808 5716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:12.715870 5716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:51:12.721454 5716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:39299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:12.722832 5716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:12.724236 5716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:12.735970 5723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:12.735955 5728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:12.735955 5725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:12.736406 5716 server_base.cc:1034] running on GCE node
I20260810 01:51:12.736946 5716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:12.737726 5716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:12.738906 5716 hybrid_clock.cc:648] HybridClock initialized: now 1786326672738883 us; error 34 us; skew 500 ppm
Aug 10 01:51:12 dist-test-slave-kvfs krb5kdc[5168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326672, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:12.742339 5716 init.cc:377] Logged in from keytab as kudu/127.9.77.131@KRBTEST.COM (short username kudu)
I20260810 01:51:12.743479 5716 webserver.cc:460] Webserver started at http://127.9.77.131:37677/ using document root <none> and password file <none>
I20260810 01:51:12.744292 5716 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:12.744398 5716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:12.744681 5716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:12.747195 5716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-2/data/instance:
uuid: "976610bdfd824f239e0f0200a8cf6da7"
format_stamp: "Formatted at 2026-08-10 01:51:12 on dist-test-slave-kvfs"
server_key: "f3236b460b243c387553f918a7297a63"
server_key_iv: "8d5900b34ecbf9f2703ce8df0f6be291"
server_key_version: "encryptionkey@0"
I20260810 01:51:12.747942 5716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance:
uuid: "976610bdfd824f239e0f0200a8cf6da7"
format_stamp: "Formatted at 2026-08-10 01:51:12 on dist-test-slave-kvfs"
server_key: "f3236b460b243c387553f918a7297a63"
server_key_iv: "8d5900b34ecbf9f2703ce8df0f6be291"
server_key_version: "encryptionkey@0"
I20260810 01:51:12.752528 5716 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 01:51:12.755461 5736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:12.756978 5716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:51:12.757133 5716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "976610bdfd824f239e0f0200a8cf6da7"
format_stamp: "Formatted at 2026-08-10 01:51:12 on dist-test-slave-kvfs"
server_key: "f3236b460b243c387553f918a7297a63"
server_key_iv: "8d5900b34ecbf9f2703ce8df0f6be291"
server_key_version: "encryptionkey@0"
I20260810 01:51:12.757266 5716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:12.771493 5716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:12.775036 5716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:12.775269 5716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:12.775985 5716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:51:12.777041 5716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:51:12.777117 5716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:12.777196 5716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:51:12.777252 5716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:12.788137 5716 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:39751
I20260810 01:51:12.788160 5896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:39751 every 8 connection(s)
I20260810 01:51:12.789283 5716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:51:12.794844 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 5716
I20260810 01:51:12.795013 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance
I20260810 01:51:12.795348 9526 external_mini_cluster.cc:1428] Setting key d909416c210e16125f79d3328d035049
Aug 10 01:51:12 dist-test-slave-kvfs krb5kdc[5168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326672, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:51:12.804957 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:12.791467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:52267 (local address 127.9.77.190:39299)
0810 01:51:12.791635 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:12.791639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:12.792710 (+ 1071us) server_negotiation.cc:407] Connection header received
0810 01:51:12.794305 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:12.794309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:12.794384 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:12.794501 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:12.796167 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:12.796861 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:12.798313 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:12.798555 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:12.798672 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:12.801321 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:12.801330 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:12.801332 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:12.801359 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:12.802676 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:12.803214 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:12.803222 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:12.803224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:12.803307 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:12.803679 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:12.803685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:12.803691 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:12.803898 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:12.804022 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:12.804612 (+ 590us) server_negotiation.cc:299] Negotiation successful
0810 01:51:12.804748 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 01:51:12.805939 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:12.791938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:39299 (local address 127.9.77.131:52267)
0810 01:51:12.792537 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:12.792586 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:12.794060 (+ 1474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:12.794707 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:12.794715 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:12.795241 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:12.795996 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:12.796037 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:12.796995 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:12.796999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:12.798187 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:12.798197 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:12.798715 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:12.798719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:12.798912 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:12.799540 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:51:12.799557 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:12.801148 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:12.802798 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:12.802805 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:12.802824 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:12.803100 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:12.803403 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:12.803407 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:12.803409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:12.803568 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:12.804017 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:12.804060 (+ 43us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:12.804346 (+ 286us) client_negotiation.cc:769] Sending connection context
0810 01:51:12.804552 (+ 206us) client_negotiation.cc:240] Negotiation successful
0810 01:51:12.804917 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":144,"threads_started":1}
I20260810 01:51:12.807219 5897 heartbeater.cc:347] Connected to a master server at 127.9.77.190:39299
I20260810 01:51:12.807492 5897 heartbeater.cc:464] Registering TS with master...
I20260810 01:51:12.808097 5897 heartbeater.cc:511] Master 127.9.77.190:39299 requested a full tablet report, sending...
I20260810 01:51:12.809458 5244 ts_manager.cc:194] Registered new tserver with Master: 976610bdfd824f239e0f0200a8cf6da7 (127.9.77.131:39751)
I20260810 01:51:12.810022 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 01:51:12.810204 5244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.131@KRBTEST.COM'} at 127.9.77.131:52267
I20260810 01:51:12.820540 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:12.812790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40764 (local address 127.9.77.190:39299)
0810 01:51:12.812915 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:12.812918 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:12.813176 (+ 258us) server_negotiation.cc:407] Connection header received
0810 01:51:12.813288 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:12.813290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:12.813331 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:12.813392 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:12.814123 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:12.814629 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:12.815472 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:12.815683 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:12.815757 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:12.817415 (+ 1658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:12.817418 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:12.817419 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:12.817439 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:12.818656 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:12.819145 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:12.819147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:12.819148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:12.819225 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:12.819557 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:12.819560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:12.819561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:12.819825 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:12.819978 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:12.820316 (+ 338us) server_negotiation.cc:299] Negotiation successful
0810 01:51:12.820408 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 01:51:12.823959 5244 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 01:51:12.827692 5244 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:51:12.846194 5920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:12.840491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39722 (local address 127.9.77.131:39751)
0810 01:51:12.840923 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:12.840938 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:12.841474 (+ 536us) server_negotiation.cc:407] Connection header received
0810 01:51:12.841577 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:12.841582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:12.841772 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:12.841951 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:12.842259 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:12.843217 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:12.844509 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:12.845415 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:12.845549 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:12.845744 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:12.845884 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 01:51:12.845996 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":175,"threads_started":1}
I20260810 01:51:12.846186 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:12.839911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:39751 (local address 127.0.0.1:39722)
0810 01:51:12.840905 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:12.840922 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:12.841035 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:12.841967 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:12.841971 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:12.841992 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:12.842123 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:12.842130 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:12.843358 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:12.843362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:12.844380 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:12.844390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:12.845549 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:12.845553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:12.845676 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:12.845719 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 01:51:12.845849 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 01:51:12.846001 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":31,"thread_start_us":173,"threads_started":1}
I20260810 01:51:12.846772 5916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:12.838625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:37217 (local address 127.0.0.1:40884)
0810 01:51:12.839814 (+ 1189us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:12.839880 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:12.840093 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:12.841083 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:12.841086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:12.841098 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:12.841357 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:12.841362 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:12.842892 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:12.842895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:12.844723 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:12.844737 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:12.846323 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:12.846327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:12.846434 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:12.846458 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 01:51:12.846528 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 01:51:12.846612 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":935,"mutex_wait_us":218,"thread_start_us":108,"threads_started":1}
I20260810 01:51:12.847121 5919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:12.838736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:44989 (local address 127.0.0.1:34196)
0810 01:51:12.840351 (+ 1615us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:12.840378 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:12.840497 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:12.841050 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:12.841060 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:12.841087 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:12.841353 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:12.841361 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:12.842806 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:12.842813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:12.845009 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:12.845023 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:12.846660 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:12.846664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:12.846769 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:12.846805 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 01:51:12.846881 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 01:51:12.846980 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1438,"mutex_wait_us":235,"thread_start_us":126,"threads_started":1}
I20260810 01:51:12.847406 5917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:12.838807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40884 (local address 127.9.77.129:37217)
0810 01:51:12.840241 (+ 1434us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:12.840247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:12.840276 (+ 29us) server_negotiation.cc:407] Connection header received
0810 01:51:12.840377 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:12.840384 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:12.840711 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:12.840894 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:12.841503 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:12.842777 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:12.844937 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:12.845941 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:12.846063 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:12.846250 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:12.847053 (+ 803us) server_negotiation.cc:299] Negotiation successful
0810 01:51:12.847184 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1307,"thread_start_us":166,"threads_started":1}
I20260810 01:51:12.847955 5918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:12.839581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34196 (local address 127.9.77.130:44989)
0810 01:51:12.839968 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:12.839988 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:12.840481 (+ 493us) server_negotiation.cc:407] Connection header received
0810 01:51:12.840627 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:12.840635 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:12.840869 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:12.841236 (+ 367us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:12.841470 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:12.842650 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:12.845162 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:12.846146 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:12.846283 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:12.847494 (+ 1211us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:12.847555 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 01:51:12.847665 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":151,"threads_started":1}
I20260810 01:51:12.850697 5614 tablet_service.cc:1467] Processing CreateTablet for tablet 41203a102b654d02ad9a6cc6088cb72f (DEFAULT_TABLE table=test-table [id=beb197f8feef43ee816fe982921f833b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:51:12.850703 5425 tablet_service.cc:1467] Processing CreateTablet for tablet 41203a102b654d02ad9a6cc6088cb72f (DEFAULT_TABLE table=test-table [id=beb197f8feef43ee816fe982921f833b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:51:12.850703 5803 tablet_service.cc:1467] Processing CreateTablet for tablet 41203a102b654d02ad9a6cc6088cb72f (DEFAULT_TABLE table=test-table [id=beb197f8feef43ee816fe982921f833b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:51:12.852011 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41203a102b654d02ad9a6cc6088cb72f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:12.852003 5614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41203a102b654d02ad9a6cc6088cb72f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:12.852103 5803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41203a102b654d02ad9a6cc6088cb72f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:12.860235 5921 tablet_bootstrap.cc:492] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe: Bootstrap starting.
I20260810 01:51:12.861416 5924 tablet_bootstrap.cc:492] T 41203a102b654d02ad9a6cc6088cb72f P 976610bdfd824f239e0f0200a8cf6da7: Bootstrap starting.
I20260810 01:51:12.863103 5921 tablet_bootstrap.cc:654] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:12.863565 5924 tablet_bootstrap.cc:654] T 41203a102b654d02ad9a6cc6088cb72f P 976610bdfd824f239e0f0200a8cf6da7: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:12.864486 5922 tablet_bootstrap.cc:492] T 41203a102b654d02ad9a6cc6088cb72f P 93e19962e32046cab363a72ec831cbc7: Bootstrap starting.
I20260810 01:51:12.864617 5921 log.cc:824] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:12.865087 5924 log.cc:824] T 41203a102b654d02ad9a6cc6088cb72f P 976610bdfd824f239e0f0200a8cf6da7: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:12.867175 5921 tablet_bootstrap.cc:492] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe: No bootstrap required, opened a new log
I20260810 01:51:12.867192 5924 tablet_bootstrap.cc:492] T 41203a102b654d02ad9a6cc6088cb72f P 976610bdfd824f239e0f0200a8cf6da7: No bootstrap required, opened a new log
I20260810 01:51:12.867340 5922 tablet_bootstrap.cc:654] T 41203a102b654d02ad9a6cc6088cb72f P 93e19962e32046cab363a72ec831cbc7: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:12.867408 5924 ts_tablet_manager.cc:1397] T 41203a102b654d02ad9a6cc6088cb72f P 976610bdfd824f239e0f0200a8cf6da7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260810 01:51:12.867408 5921 ts_tablet_manager.cc:1397] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 01:51:12.868253 5922 log.cc:824] T 41203a102b654d02ad9a6cc6088cb72f P 93e19962e32046cab363a72ec831cbc7: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:12.870211 5922 tablet_bootstrap.cc:492] T 41203a102b654d02ad9a6cc6088cb72f P 93e19962e32046cab363a72ec831cbc7: No bootstrap required, opened a new log
I20260810 01:51:12.870447 5922 ts_tablet_manager.cc:1397] T 41203a102b654d02ad9a6cc6088cb72f P 93e19962e32046cab363a72ec831cbc7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 01:51:12.871852 5924 raft_consensus.cc:348] T 41203a102b654d02ad9a6cc6088cb72f P 976610bdfd824f239e0f0200a8cf6da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e19962e32046cab363a72ec831cbc7" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37217 } } peers { permanent_uuid: "77c2879ddf82486aafb429134defbdbe" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44989 } } peers { permanent_uuid: "976610bdfd824f239e0f0200a8cf6da7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39751 } }
I20260810 01:51:12.871852 5921 raft_consensus.cc:348] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e19962e32046cab363a72ec831cbc7" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37217 } } peers { permanent_uuid: "77c2879ddf82486aafb429134defbdbe" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44989 } } peers { permanent_uuid: "976610bdfd824f239e0f0200a8cf6da7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39751 } }
I20260810 01:51:12.872228 5921 raft_consensus.cc:374] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:12.872228 5924 raft_consensus.cc:374] T 41203a102b654d02ad9a6cc6088cb72f P 976610bdfd824f239e0f0200a8cf6da7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:12.872294 5924 raft_consensus.cc:729] T 41203a102b654d02ad9a6cc6088cb72f P 976610bdfd824f239e0f0200a8cf6da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 976610bdfd824f239e0f0200a8cf6da7, State: Initialized, Role: FOLLOWER
I20260810 01:51:12.872304 5921 raft_consensus.cc:729] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c2879ddf82486aafb429134defbdbe, State: Initialized, Role: FOLLOWER
I20260810 01:51:12.872859 5921 consensus_queue.cc:260] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e19962e32046cab363a72ec831cbc7" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37217 } } peers { permanent_uuid: "77c2879ddf82486aafb429134defbdbe" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44989 } } peers { permanent_uuid: "976610bdfd824f239e0f0200a8cf6da7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39751 } }
I20260810 01:51:12.872859 5924 consensus_queue.cc:260] T 41203a102b654d02ad9a6cc6088cb72f P 976610bdfd824f239e0f0200a8cf6da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e19962e32046cab363a72ec831cbc7" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37217 } } peers { permanent_uuid: "77c2879ddf82486aafb429134defbdbe" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44989 } } peers { permanent_uuid: "976610bdfd824f239e0f0200a8cf6da7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39751 } }
I20260810 01:51:12.873965 5707 heartbeater.cc:503] Master 127.9.77.190:39299 was elected leader, sending a full tablet report...
I20260810 01:51:12.874078 5921 ts_tablet_manager.cc:1428] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 01:51:12.874078 5924 ts_tablet_manager.cc:1428] T 41203a102b654d02ad9a6cc6088cb72f P 976610bdfd824f239e0f0200a8cf6da7: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 01:51:12.874596 5897 heartbeater.cc:503] Master 127.9.77.190:39299 was elected leader, sending a full tablet report...
I20260810 01:51:12.875448 5922 raft_consensus.cc:348] T 41203a102b654d02ad9a6cc6088cb72f P 93e19962e32046cab363a72ec831cbc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e19962e32046cab363a72ec831cbc7" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37217 } } peers { permanent_uuid: "77c2879ddf82486aafb429134defbdbe" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44989 } } peers { permanent_uuid: "976610bdfd824f239e0f0200a8cf6da7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39751 } }
I20260810 01:51:12.875773 5922 raft_consensus.cc:374] T 41203a102b654d02ad9a6cc6088cb72f P 93e19962e32046cab363a72ec831cbc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:12.875844 5922 raft_consensus.cc:729] T 41203a102b654d02ad9a6cc6088cb72f P 93e19962e32046cab363a72ec831cbc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93e19962e32046cab363a72ec831cbc7, State: Initialized, Role: FOLLOWER
I20260810 01:51:12.876446 5922 consensus_queue.cc:260] T 41203a102b654d02ad9a6cc6088cb72f P 93e19962e32046cab363a72ec831cbc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e19962e32046cab363a72ec831cbc7" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37217 } } peers { permanent_uuid: "77c2879ddf82486aafb429134defbdbe" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44989 } } peers { permanent_uuid: "976610bdfd824f239e0f0200a8cf6da7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39751 } }
I20260810 01:51:12.877233 5507 heartbeater.cc:503] Master 127.9.77.190:39299 was elected leader, sending a full tablet report...
I20260810 01:51:12.877804 5922 ts_tablet_manager.cc:1428] T 41203a102b654d02ad9a6cc6088cb72f P 93e19962e32046cab363a72ec831cbc7: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 01:51:13.007894 5932 raft_consensus.cc:482] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:51:13.008281 5932 raft_consensus.cc:504] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e19962e32046cab363a72ec831cbc7" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37217 } } peers { permanent_uuid: "77c2879ddf82486aafb429134defbdbe" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44989 } } peers { permanent_uuid: "976610bdfd824f239e0f0200a8cf6da7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39751 } }
I20260810 01:51:13.009915 5932 leader_election.cc:288] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93e19962e32046cab363a72ec831cbc7 (127.9.77.129:37217), 976610bdfd824f239e0f0200a8cf6da7 (127.9.77.131:39751)
I20260810 01:51:13.015919 5917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:13.010520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:60249 (local address 127.9.77.129:37217)
0810 01:51:13.010707 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:13.010713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:13.010732 (+ 19us) server_negotiation.cc:407] Connection header received
0810 01:51:13.010832 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:13.010835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:13.010925 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:13.011007 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:13.011407 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:13.012319 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:13.014534 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:13.015201 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:13.015326 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:13.015396 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:13.015673 (+ 277us) server_negotiation.cc:299] Negotiation successful
0810 01:51:13.015761 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260810 01:51:13.015919 5920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:13.010454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:53171 (local address 127.9.77.131:39751)
0810 01:51:13.010709 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:13.010715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:13.010802 (+ 87us) server_negotiation.cc:407] Connection header received
0810 01:51:13.010944 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:13.010950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:13.011018 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:13.011098 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:13.011465 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:13.012318 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:13.014534 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:13.015188 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:13.015326 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:13.015396 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:13.015673 (+ 277us) server_negotiation.cc:299] Negotiation successful
0810 01:51:13.015755 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260810 01:51:13.015954 5709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:13.010191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:37217 (local address 127.9.77.130:60249)
0810 01:51:13.010375 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:13.010396 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:13.010519 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:13.011020 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:13.011022 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:13.011037 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:13.011251 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:13.011263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:13.012478 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:13.012483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:13.014382 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:13.014397 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:13.015338 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:13.015343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:13.015496 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:13.015547 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 01:51:13.015642 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 01:51:13.015740 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":315}
I20260810 01:51:13.015964 5935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:13.010317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:39751 (local address 127.9.77.130:53171)
0810 01:51:13.010686 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:13.010701 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:13.010835 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:13.011126 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:13.011129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:13.011143 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:13.011315 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:13.011325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:13.012478 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:13.012483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:13.014382 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:13.014397 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:13.015338 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:13.015343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:13.015490 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:13.015547 (+ 57us) client_negotiation.cc:769] Sending connection context
0810 01:51:13.015642 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 01:51:13.015750 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":269,"thread_start_us":131,"threads_started":1}
I20260810 01:51:13.016916 5450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41203a102b654d02ad9a6cc6088cb72f" candidate_uuid: "77c2879ddf82486aafb429134defbdbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e19962e32046cab363a72ec831cbc7" is_pre_election: true
I20260810 01:51:13.016942 5828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41203a102b654d02ad9a6cc6088cb72f" candidate_uuid: "77c2879ddf82486aafb429134defbdbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "976610bdfd824f239e0f0200a8cf6da7" is_pre_election: true
I20260810 01:51:13.017354 5450 raft_consensus.cc:2454] T 41203a102b654d02ad9a6cc6088cb72f P 93e19962e32046cab363a72ec831cbc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77c2879ddf82486aafb429134defbdbe in term 0.
I20260810 01:51:13.017354 5828 raft_consensus.cc:2454] T 41203a102b654d02ad9a6cc6088cb72f P 976610bdfd824f239e0f0200a8cf6da7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77c2879ddf82486aafb429134defbdbe in term 0.
W20260810 01:51:13.017386 5509 tablet.cc:2369] T 41203a102b654d02ad9a6cc6088cb72f P 93e19962e32046cab363a72ec831cbc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 01:51:13.018021 5544 leader_election.cc:302] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 77c2879ddf82486aafb429134defbdbe, 93e19962e32046cab363a72ec831cbc7, 976610bdfd824f239e0f0200a8cf6da7; no voters:
I20260810 01:51:13.018362 5932 raft_consensus.cc:2781] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:51:13.018447 5932 raft_consensus.cc:482] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:51:13.018502 5932 raft_consensus.cc:3037] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:13.019778 5932 raft_consensus.cc:504] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e19962e32046cab363a72ec831cbc7" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37217 } } peers { permanent_uuid: "77c2879ddf82486aafb429134defbdbe" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44989 } } peers { permanent_uuid: "976610bdfd824f239e0f0200a8cf6da7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39751 } }
I20260810 01:51:13.020269 5932 leader_election.cc:288] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe [CANDIDATE]: Term 1 election: Requested vote from peers 93e19962e32046cab363a72ec831cbc7 (127.9.77.129:37217), 976610bdfd824f239e0f0200a8cf6da7 (127.9.77.131:39751)
I20260810 01:51:13.020707 5828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41203a102b654d02ad9a6cc6088cb72f" candidate_uuid: "77c2879ddf82486aafb429134defbdbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "976610bdfd824f239e0f0200a8cf6da7"
I20260810 01:51:13.020711 5450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41203a102b654d02ad9a6cc6088cb72f" candidate_uuid: "77c2879ddf82486aafb429134defbdbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e19962e32046cab363a72ec831cbc7"
I20260810 01:51:13.020819 5828 raft_consensus.cc:3037] T 41203a102b654d02ad9a6cc6088cb72f P 976610bdfd824f239e0f0200a8cf6da7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:13.020887 5450 raft_consensus.cc:3037] T 41203a102b654d02ad9a6cc6088cb72f P 93e19962e32046cab363a72ec831cbc7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:13.022042 5828 raft_consensus.cc:2454] T 41203a102b654d02ad9a6cc6088cb72f P 976610bdfd824f239e0f0200a8cf6da7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77c2879ddf82486aafb429134defbdbe in term 1.
I20260810 01:51:13.022136 5450 raft_consensus.cc:2454] T 41203a102b654d02ad9a6cc6088cb72f P 93e19962e32046cab363a72ec831cbc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77c2879ddf82486aafb429134defbdbe in term 1.
I20260810 01:51:13.022435 5544 leader_election.cc:302] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77c2879ddf82486aafb429134defbdbe, 976610bdfd824f239e0f0200a8cf6da7; no voters:
I20260810 01:51:13.022657 5932 raft_consensus.cc:2781] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:13.022912 5932 raft_consensus.cc:686] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe [term 1 LEADER]: Becoming Leader. State: Replica: 77c2879ddf82486aafb429134defbdbe, State: Running, Role: LEADER
I20260810 01:51:13.023301 5932 consensus_queue.cc:237] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e19962e32046cab363a72ec831cbc7" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37217 } } peers { permanent_uuid: "77c2879ddf82486aafb429134defbdbe" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44989 } } peers { permanent_uuid: "976610bdfd824f239e0f0200a8cf6da7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39751 } }
I20260810 01:51:13.027279 5242 catalog_manager.cc:5495] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe reported cstate change: term changed from 0 to 1, leader changed from <none> to 77c2879ddf82486aafb429134defbdbe (127.9.77.130). New cstate: current_term: 1 leader_uuid: "77c2879ddf82486aafb429134defbdbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93e19962e32046cab363a72ec831cbc7" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77c2879ddf82486aafb429134defbdbe" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 44989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "976610bdfd824f239e0f0200a8cf6da7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39751 } health_report { overall_health: UNKNOWN } } }
W20260810 01:51:13.031790 5708 tablet.cc:2369] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 01:51:13.040382 5900 tablet.cc:2369] T 41203a102b654d02ad9a6cc6088cb72f P 976610bdfd824f239e0f0200a8cf6da7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 01:51:13.084282 5918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:13.080282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34204 (local address 127.9.77.130:44989)
0810 01:51:13.080445 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:13.080449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:13.080467 (+ 18us) server_negotiation.cc:407] Connection header received
0810 01:51:13.080563 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:13.080566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:13.080632 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:13.080735 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:51:13.081037 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:13.081640 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:13.082594 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:13.082831 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:13.082919 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:13.083232 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:51:13.083822 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:51:13.083953 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:13.084076 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 01:51:13.084128 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 01:51:13.092178 5828 raft_consensus.cc:1261] T 41203a102b654d02ad9a6cc6088cb72f P 976610bdfd824f239e0f0200a8cf6da7 [term 1 FOLLOWER]: Refusing update from remote peer 77c2879ddf82486aafb429134defbdbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:51:13.092177 5450 raft_consensus.cc:1261] T 41203a102b654d02ad9a6cc6088cb72f P 93e19962e32046cab363a72ec831cbc7 [term 1 FOLLOWER]: Refusing update from remote peer 77c2879ddf82486aafb429134defbdbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:51:13.093080 5936 consensus_queue.cc:1035] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "976610bdfd824f239e0f0200a8cf6da7" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 39751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:51:13.093407 5932 consensus_queue.cc:1035] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e19962e32046cab363a72ec831cbc7" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:51:13.101794 5940 mvcc.cc:205] Tried to move back new op lower bound from 7316794052975857664 to 7316794052706885632. Current Snapshot: MvccSnapshot[applied={T|T < 7316794052975857664}]
I20260810 01:51:13.102044 5945 mvcc.cc:205] Tried to move back new op lower bound from 7316794052975857664 to 7316794052706885632. Current Snapshot: MvccSnapshot[applied={T|T < 7316794052975857664}]
I20260810 01:51:13.113021 5244 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:51:13.113283 5244 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:51:13.116276 5244 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e879a69b4c8e44a1a6ae1622dadb405a: Sending DeleteTablet for 3 replicas of tablet 41203a102b654d02ad9a6cc6088cb72f
I20260810 01:51:13.117267 5614 tablet_service.cc:1514] Processing DeleteTablet for tablet 41203a102b654d02ad9a6cc6088cb72f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:51:13 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:34196
I20260810 01:51:13.117270 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet 41203a102b654d02ad9a6cc6088cb72f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:51:13 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:40884
I20260810 01:51:13.117378 5803 tablet_service.cc:1514] Processing DeleteTablet for tablet 41203a102b654d02ad9a6cc6088cb72f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:51:13 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:39722
I20260810 01:51:13.117918 5953 tablet_replica.cc:330] T 41203a102b654d02ad9a6cc6088cb72f P 93e19962e32046cab363a72ec831cbc7: stopping tablet replica
I20260810 01:51:13.118346 5953 raft_consensus.cc:2229] T 41203a102b654d02ad9a6cc6088cb72f P 93e19962e32046cab363a72ec831cbc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:51:13.118769 5953 raft_consensus.cc:2258] T 41203a102b654d02ad9a6cc6088cb72f P 93e19962e32046cab363a72ec831cbc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:51:13.118816 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 5334
I20260810 01:51:13.119273 5954 tablet_replica.cc:330] T 41203a102b654d02ad9a6cc6088cb72f P 976610bdfd824f239e0f0200a8cf6da7: stopping tablet replica
I20260810 01:51:13.119635 5954 raft_consensus.cc:2229] T 41203a102b654d02ad9a6cc6088cb72f P 976610bdfd824f239e0f0200a8cf6da7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:51:13.120002 5954 raft_consensus.cc:2258] T 41203a102b654d02ad9a6cc6088cb72f P 976610bdfd824f239e0f0200a8cf6da7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:51:13.133736 5954 ts_tablet_manager.cc:1905] T 41203a102b654d02ad9a6cc6088cb72f P 976610bdfd824f239e0f0200a8cf6da7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:51:13.132010 5952 tablet_replica.cc:330] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe: stopping tablet replica
I20260810 01:51:13.136821 5952 raft_consensus.cc:2229] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe [term 1 LEADER]: Raft consensus shutting down.
I20260810 01:51:13.137107 5954 ts_tablet_manager.cc:1918] T 41203a102b654d02ad9a6cc6088cb72f P 976610bdfd824f239e0f0200a8cf6da7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:51:13.137246 5954 log.cc:1196] T 41203a102b654d02ad9a6cc6088cb72f P 976610bdfd824f239e0f0200a8cf6da7: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786326392877349-9526-0/minicluster-data/ts-2/wal/wals/41203a102b654d02ad9a6cc6088cb72f
I20260810 01:51:13.137663 5954 ts_tablet_manager.cc:1939] T 41203a102b654d02ad9a6cc6088cb72f P 976610bdfd824f239e0f0200a8cf6da7: Deleting consensus metadata
I20260810 01:51:13.137997 5952 raft_consensus.cc:2258] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:51:13.139794 5217 catalog_manager.cc:4841] TS 976610bdfd824f239e0f0200a8cf6da7 (127.9.77.131:39751): tablet 41203a102b654d02ad9a6cc6088cb72f (table test-table [id=beb197f8feef43ee816fe982921f833b]) successfully deleted
I20260810 01:51:13.140431 5952 ts_tablet_manager.cc:1905] T 41203a102b654d02ad9a6cc6088cb72f P 77c2879ddf82486aafb429134defbdbe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 01:51:13.140681 5220 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.77.129:37217 (error 108)
I20260810 01:51:13.142123 5916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:13.141716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:37217 (local address 127.0.0.1:40898)
0810 01:51:13.141892 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:13.141986 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.129:37217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 01:51:13.142256 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 5516
W20260810 01:51:13.168596 5220 catalog_manager.cc:4584] TS 93e19962e32046cab363a72ec831cbc7 (127.9.77.129:37217): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 41203a102b654d02ad9a6cc6088cb72f: Network error: Client connection negotiation failed: client connection to 127.9.77.129:37217: connect: Connection refused (error 111)
I20260810 01:51:13.175944 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 5716
I20260810 01:51:13.185842 5916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:13.176377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:44989 (local address 127.0.0.1:34218)
0810 01:51:13.185497 (+ 9120us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:13.185641 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.130:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8909}
I20260810 01:51:13.195178 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 5188
2026-08-10T01:51:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2952 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T01:51:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:51:13Z Disabled control of system clock
I20260810 01:51:13.251348 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:45181
--webserver_interface=127.9.77.190
--webserver_port=0
--builtin_ntp_servers=127.9.77.148:35685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:45181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 01:51:13.385634 5960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:13.386008 5960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:13.386046 5960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:13.390287 5960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:13.390367 5960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:13.390381 5960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:13.390399 5960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:13.390415 5960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:13.396241 5960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:45181
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:45181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:13.397742 5960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:13.399571 5960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:13.409729 5965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:13.409749 5972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:13.409919 5960 server_base.cc:1034] running on GCE node
W20260810 01:51:13.409775 5968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:13.410776 5960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:13.412068 5960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:13.413262 5960 hybrid_clock.cc:648] HybridClock initialized: now 1786326673413238 us; error 38 us; skew 500 ppm
I20260810 01:51:13.415438 5960 webserver.cc:460] Webserver started at http://127.9.77.190:39145/ using document root <none> and password file <none>
I20260810 01:51:13.416160 5960 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:13.416235 5960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:13.416460 5960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:13.418982 5960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "f7f7c9a1cd014b6db6759725a319226c"
format_stamp: "Formatted at 2026-08-10 01:51:13 on dist-test-slave-kvfs"
server_key: "905c4d604e8c63bac738bd5bba5531d0"
server_key_iv: "4458ec0d1b96b983d7682a4e715fa62f"
server_key_version: "encryptionkey@0"
I20260810 01:51:13.419591 5960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "f7f7c9a1cd014b6db6759725a319226c"
format_stamp: "Formatted at 2026-08-10 01:51:13 on dist-test-slave-kvfs"
server_key: "905c4d604e8c63bac738bd5bba5531d0"
server_key_iv: "4458ec0d1b96b983d7682a4e715fa62f"
server_key_version: "encryptionkey@0"
I20260810 01:51:13.423853 5960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 01:51:13.426612 5981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:13.428138 5960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:51:13.428260 5960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "f7f7c9a1cd014b6db6759725a319226c"
format_stamp: "Formatted at 2026-08-10 01:51:13 on dist-test-slave-kvfs"
server_key: "905c4d604e8c63bac738bd5bba5531d0"
server_key_iv: "4458ec0d1b96b983d7682a4e715fa62f"
server_key_version: "encryptionkey@0"
I20260810 01:51:13.428362 5960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:13.446273 5960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:13.447124 5960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:13.447304 5960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:13.455921 5960 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:45181
I20260810 01:51:13.455936 6071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:45181 every 8 connection(s)
I20260810 01:51:13.457180 5960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:51:13.459619 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 5960
I20260810 01:51:13.459800 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:51:13.460260 9526 external_mini_cluster.cc:1428] Setting key ba76674a64a64990ed129771907f1bfa
I20260810 01:51:13.461752 6072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:13.470273 6072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7f7c9a1cd014b6db6759725a319226c: Bootstrap starting.
I20260810 01:51:13.473115 6076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:13.462122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35060 (local address 127.9.77.190:45181)
0810 01:51:13.462992 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:13.463018 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:13.463073 (+ 55us) server_negotiation.cc:407] Connection header received
0810 01:51:13.464182 (+ 1109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:13.464211 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:13.464834 (+ 623us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:13.465288 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:13.465890 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:13.467522 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:13.468591 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:13.469346 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:13.469530 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:13.470514 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:13.470520 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:13.470538 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:13.471126 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:13.471213 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:13.471244 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:13.471351 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:13.471542 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 01:51:13.471907 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":171,"threads_started":1}
I20260810 01:51:13.473802 6072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7f7c9a1cd014b6db6759725a319226c: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:13.474838 6072 log.cc:824] T 00000000000000000000000000000000 P f7f7c9a1cd014b6db6759725a319226c: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:13.477437 6072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7f7c9a1cd014b6db6759725a319226c: No bootstrap required, opened a new log
I20260810 01:51:13.481786 6072 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7f7c9a1cd014b6db6759725a319226c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f7c9a1cd014b6db6759725a319226c" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45181 } }
I20260810 01:51:13.482108 6072 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7f7c9a1cd014b6db6759725a319226c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:13.482200 6072 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7f7c9a1cd014b6db6759725a319226c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f7c9a1cd014b6db6759725a319226c, State: Initialized, Role: FOLLOWER
I20260810 01:51:13.482766 6072 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7f7c9a1cd014b6db6759725a319226c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f7c9a1cd014b6db6759725a319226c" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45181 } }
I20260810 01:51:13.482987 6072 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7f7c9a1cd014b6db6759725a319226c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:51:13.483085 6072 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7f7c9a1cd014b6db6759725a319226c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:51:13.483206 6072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7f7c9a1cd014b6db6759725a319226c [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:13.484509 6072 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7f7c9a1cd014b6db6759725a319226c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f7c9a1cd014b6db6759725a319226c" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45181 } }
I20260810 01:51:13.484941 6072 leader_election.cc:302] T 00000000000000000000000000000000 P f7f7c9a1cd014b6db6759725a319226c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7f7c9a1cd014b6db6759725a319226c; no voters:
I20260810 01:51:13.485317 6072 leader_election.cc:288] T 00000000000000000000000000000000 P f7f7c9a1cd014b6db6759725a319226c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:51:13.485484 6080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7f7c9a1cd014b6db6759725a319226c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:13.485757 6080 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7f7c9a1cd014b6db6759725a319226c [term 1 LEADER]: Becoming Leader. State: Replica: f7f7c9a1cd014b6db6759725a319226c, State: Running, Role: LEADER
I20260810 01:51:13.486132 6080 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7f7c9a1cd014b6db6759725a319226c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f7c9a1cd014b6db6759725a319226c" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45181 } }
I20260810 01:51:13.486858 6072 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7f7c9a1cd014b6db6759725a319226c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:13.488245 6081 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7f7c9a1cd014b6db6759725a319226c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7f7c9a1cd014b6db6759725a319226c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f7c9a1cd014b6db6759725a319226c" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45181 } } }
I20260810 01:51:13.488404 6081 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7f7c9a1cd014b6db6759725a319226c [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:13.488768 6082 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7f7c9a1cd014b6db6759725a319226c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7f7c9a1cd014b6db6759725a319226c. Latest consensus state: current_term: 1 leader_uuid: "f7f7c9a1cd014b6db6759725a319226c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f7c9a1cd014b6db6759725a319226c" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45181 } } }
I20260810 01:51:13.488888 6082 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7f7c9a1cd014b6db6759725a319226c [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:13.488869 6089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:51:13.492873 6089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:51:13.500537 6089 catalog_manager.cc:1347] Generated new cluster ID: bd138f8c88c942278ae30393fd9404c4
I20260810 01:51:13.500623 6089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:51:13.518464 6089 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:51:13.519701 6089 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:51:13.533072 6089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7f7c9a1cd014b6db6759725a319226c: Generated new TSK 0
I20260810 01:51:13.534170 6089 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:51:13.540073 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:0
--local_ip_for_outbound_sockets=127.9.77.129
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:45181
--builtin_ntp_servers=127.9.77.148:35685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 01:51:13.679306 6107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:13.679741 6107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:13.679821 6107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:13.684067 6107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:13.684142 6107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:13.684201 6107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:51:13.689543 6107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:45181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:13.691044 6107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:13.692881 6107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:13.704860 6116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:13.704844 6115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:13.705044 6107 server_base.cc:1034] running on GCE node
W20260810 01:51:13.704844 6120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:13.705762 6107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:13.706614 6107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:13.707819 6107 hybrid_clock.cc:648] HybridClock initialized: now 1786326673707794 us; error 37 us; skew 500 ppm
I20260810 01:51:13.710254 6107 webserver.cc:460] Webserver started at http://127.9.77.129:46019/ using document root <none> and password file <none>
I20260810 01:51:13.710992 6107 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:13.711093 6107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:13.711381 6107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:13.714020 6107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-0/data/instance:
uuid: "b35ec97c5f9b4e6d9222dc0e2a010ba6"
format_stamp: "Formatted at 2026-08-10 01:51:13 on dist-test-slave-kvfs"
server_key: "c55783918629836d462d3186522b0e6f"
server_key_iv: "f87cc0100db233410f38c70c8cb970bc"
server_key_version: "encryptionkey@0"
I20260810 01:51:13.714684 6107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance:
uuid: "b35ec97c5f9b4e6d9222dc0e2a010ba6"
format_stamp: "Formatted at 2026-08-10 01:51:13 on dist-test-slave-kvfs"
server_key: "c55783918629836d462d3186522b0e6f"
server_key_iv: "f87cc0100db233410f38c70c8cb970bc"
server_key_version: "encryptionkey@0"
I20260810 01:51:13.719293 6107 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 01:51:13.722498 6131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:13.724329 6107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:51:13.724488 6107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "b35ec97c5f9b4e6d9222dc0e2a010ba6"
format_stamp: "Formatted at 2026-08-10 01:51:13 on dist-test-slave-kvfs"
server_key: "c55783918629836d462d3186522b0e6f"
server_key_iv: "f87cc0100db233410f38c70c8cb970bc"
server_key_version: "encryptionkey@0"
I20260810 01:51:13.724628 6107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:13.748843 6107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:13.749845 6107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:13.750101 6107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:13.750902 6107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:51:13.752183 6107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:51:13.752264 6107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:13.752378 6107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:51:13.752434 6107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:13.763861 6107 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:37033
I20260810 01:51:13.763886 6298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:37033 every 8 connection(s)
I20260810 01:51:13.765151 6107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:51:13.771417 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 6107
I20260810 01:51:13.771575 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance
I20260810 01:51:13.771966 9526 external_mini_cluster.cc:1428] Setting key ef7da9bbac03a9476c071bac78012445
I20260810 01:51:13.774818 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:0
--local_ip_for_outbound_sockets=127.9.77.130
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:45181
--builtin_ntp_servers=127.9.77.148:35685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 01:51:13.778163 6076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:13.767724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:55885 (local address 127.9.77.190:45181)
0810 01:51:13.767933 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:13.767938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:13.769353 (+ 1415us) server_negotiation.cc:407] Connection header received
0810 01:51:13.770971 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:13.770974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:13.771053 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:13.771140 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:13.772643 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:13.773574 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:13.775467 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:13.775718 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:13.775822 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:13.777457 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:13.777475 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:13.777481 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:13.777503 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:13.777541 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:13.777556 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:13.777649 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:13.777833 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 01:51:13.777932 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 01:51:13.779124 6303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:13.768351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45181 (local address 127.9.77.129:55885)
0810 01:51:13.769163 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:13.769212 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:13.770724 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:13.771414 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:13.771425 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:13.771562 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:13.772427 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:13.772453 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:13.773754 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:13.773760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:13.775307 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:13.775322 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:13.775908 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:13.775916 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:13.776213 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:13.777068 (+ 855us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 01:51:13.777073 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:13.777146 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 01:51:13.777252 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:13.777657 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:13.777666 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:13.777671 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 01:51:13.777783 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 01:51:13.778073 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":154,"threads_started":1}
I20260810 01:51:13.780683 6299 heartbeater.cc:347] Connected to a master server at 127.9.77.190:45181
I20260810 01:51:13.780980 6299 heartbeater.cc:464] Registering TS with master...
I20260810 01:51:13.781643 6299 heartbeater.cc:511] Master 127.9.77.190:45181 requested a full tablet report, sending...
I20260810 01:51:13.783845 6012 ts_manager.cc:194] Registered new tserver with Master: b35ec97c5f9b4e6d9222dc0e2a010ba6 (127.9.77.129:37033)
I20260810 01:51:13.785962 6012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.77.129:55885
W20260810 01:51:13.919948 6304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:13.920439 6304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:13.920487 6304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:13.924906 6304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:13.925005 6304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:13.925072 6304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:51:13.930588 6304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:45181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:13.932183 6304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:13.934158 6304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:13.947088 6310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:13.947088 6313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:13.947122 6311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:13.947517 6304 server_base.cc:1034] running on GCE node
I20260810 01:51:13.948232 6304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:13.949106 6304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:13.950330 6304 hybrid_clock.cc:648] HybridClock initialized: now 1786326673950309 us; error 39 us; skew 500 ppm
I20260810 01:51:13.952826 6304 webserver.cc:460] Webserver started at http://127.9.77.130:38103/ using document root <none> and password file <none>
I20260810 01:51:13.953617 6304 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:13.953735 6304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:13.954033 6304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:13.956838 6304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-1/data/instance:
uuid: "acc648addb5b4fdcbbb6d4bf8adc7428"
format_stamp: "Formatted at 2026-08-10 01:51:13 on dist-test-slave-kvfs"
server_key: "1daf733cb5f84591506bdf782998ade8"
server_key_iv: "a05a80201152a0dc1deee73b2007208f"
server_key_version: "encryptionkey@0"
I20260810 01:51:13.957561 6304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance:
uuid: "acc648addb5b4fdcbbb6d4bf8adc7428"
format_stamp: "Formatted at 2026-08-10 01:51:13 on dist-test-slave-kvfs"
server_key: "1daf733cb5f84591506bdf782998ade8"
server_key_iv: "a05a80201152a0dc1deee73b2007208f"
server_key_version: "encryptionkey@0"
I20260810 01:51:13.962813 6304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 01:51:13.966357 6326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:13.968415 6304 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 01:51:13.968609 6304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "acc648addb5b4fdcbbb6d4bf8adc7428"
format_stamp: "Formatted at 2026-08-10 01:51:13 on dist-test-slave-kvfs"
server_key: "1daf733cb5f84591506bdf782998ade8"
server_key_iv: "a05a80201152a0dc1deee73b2007208f"
server_key_version: "encryptionkey@0"
I20260810 01:51:13.968802 6304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:14.000377 6304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:14.001312 6304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:14.001502 6304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:14.002220 6304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:51:14.003396 6304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:51:14.003448 6304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:14.003496 6304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:51:14.003517 6304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:14.014472 6304 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:34581
I20260810 01:51:14.014510 6489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:34581 every 8 connection(s)
I20260810 01:51:14.015640 6304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:51:14.017743 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 6304
I20260810 01:51:14.017903 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance
I20260810 01:51:14.018340 9526 external_mini_cluster.cc:1428] Setting key 378559169fd26fbb7a41f55203b287c2
I20260810 01:51:14.020907 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:0
--local_ip_for_outbound_sockets=127.9.77.131
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:45181
--builtin_ntp_servers=127.9.77.148:35685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 01:51:14.028318 6076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:14.018027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:51245 (local address 127.9.77.190:45181)
0810 01:51:14.018242 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:14.018248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:14.019792 (+ 1544us) server_negotiation.cc:407] Connection header received
0810 01:51:14.021710 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:14.021714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:14.021778 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:14.021873 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:14.023270 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.024206 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.025866 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.026123 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.026257 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:14.027682 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:14.027688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:14.027695 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:14.027711 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:14.027735 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:14.027744 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:14.027829 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:14.028035 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 01:51:14.028121 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 01:51:14.029553 6492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:14.018632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45181 (local address 127.9.77.130:51245)
0810 01:51:14.019598 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:14.019664 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:14.021459 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:14.022082 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:14.022091 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:14.022198 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:14.023026 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:14.023048 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.024358 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.024364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:14.025723 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:14.025735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.026308 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.026312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:14.026514 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:14.027325 (+ 811us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 01:51:14.027327 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:14.027380 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 01:51:14.027482 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:14.027831 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:14.027845 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:14.027855 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 01:51:14.028015 (+ 160us) client_negotiation.cc:240] Negotiation successful
0810 01:51:14.028403 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":179,"threads_started":1}
I20260810 01:51:14.031576 6490 heartbeater.cc:347] Connected to a master server at 127.9.77.190:45181
I20260810 01:51:14.031976 6490 heartbeater.cc:464] Registering TS with master...
I20260810 01:51:14.032809 6490 heartbeater.cc:511] Master 127.9.77.190:45181 requested a full tablet report, sending...
I20260810 01:51:14.034614 6012 ts_manager.cc:194] Registered new tserver with Master: acc648addb5b4fdcbbb6d4bf8adc7428 (127.9.77.130:34581)
I20260810 01:51:14.035709 6012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.77.130:51245
W20260810 01:51:14.162741 6493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:14.163134 6493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:14.163178 6493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:14.167537 6493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:14.167632 6493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:14.167696 6493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:51:14.173079 6493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:45181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:14.174630 6493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:14.176676 6493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:14.189152 6500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:14.189137 6504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:14.189404 6493 server_base.cc:1034] running on GCE node
W20260810 01:51:14.189126 6501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:14.190063 6493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:14.190867 6493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:14.192109 6493 hybrid_clock.cc:648] HybridClock initialized: now 1786326674192075 us; error 52 us; skew 500 ppm
I20260810 01:51:14.194397 6493 webserver.cc:460] Webserver started at http://127.9.77.131:45301/ using document root <none> and password file <none>
I20260810 01:51:14.195142 6493 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:14.195245 6493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:14.195544 6493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:14.198158 6493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-2/data/instance:
uuid: "693e202b39484507a823dd3d00bcf147"
format_stamp: "Formatted at 2026-08-10 01:51:14 on dist-test-slave-kvfs"
server_key: "8e62813ff11d9962f5462a9d499c49c4"
server_key_iv: "064182f0bbafe49730f6df9d205960bd"
server_key_version: "encryptionkey@0"
I20260810 01:51:14.198837 6493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance:
uuid: "693e202b39484507a823dd3d00bcf147"
format_stamp: "Formatted at 2026-08-10 01:51:14 on dist-test-slave-kvfs"
server_key: "8e62813ff11d9962f5462a9d499c49c4"
server_key_iv: "064182f0bbafe49730f6df9d205960bd"
server_key_version: "encryptionkey@0"
I20260810 01:51:14.203305 6493 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260810 01:51:14.206305 6516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:14.207950 6493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 01:51:14.208138 6493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "693e202b39484507a823dd3d00bcf147"
format_stamp: "Formatted at 2026-08-10 01:51:14 on dist-test-slave-kvfs"
server_key: "8e62813ff11d9962f5462a9d499c49c4"
server_key_iv: "064182f0bbafe49730f6df9d205960bd"
server_key_version: "encryptionkey@0"
I20260810 01:51:14.208288 6493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:14.245873 6493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:14.246927 6493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:14.247186 6493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:14.248111 6493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:51:14.249428 6493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:51:14.249511 6493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:14.249594 6493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:51:14.249650 6493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:14.260692 6493 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:42313
I20260810 01:51:14.260735 6681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:42313 every 8 connection(s)
I20260810 01:51:14.261941 6493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:51:14.262223 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 6493
I20260810 01:51:14.262387 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance
I20260810 01:51:14.262846 9526 external_mini_cluster.cc:1428] Setting key a448ab15db37b348df6c00b763b663ee
I20260810 01:51:14.273408 6076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:14.264554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:39839 (local address 127.9.77.190:45181)
0810 01:51:14.264745 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:14.264750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:14.266312 (+ 1562us) server_negotiation.cc:407] Connection header received
0810 01:51:14.267997 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:14.268000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:14.268123 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:14.268197 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:14.269264 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.269846 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.271104 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.271323 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.271428 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:14.272812 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:14.272815 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:14.272818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:14.272832 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:14.272857 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:14.272881 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:14.272945 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:14.273178 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 01:51:14.273247 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 01:51:14.274581 6684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:14.265234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:45181 (local address 127.9.77.131:39839)
0810 01:51:14.266134 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:14.266205 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:14.267772 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:14.268398 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:14.268405 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:14.268504 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:14.269134 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:14.269148 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.269983 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.269985 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:14.270989 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:14.270999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.271504 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.271508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:14.271689 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:14.272472 (+ 783us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 01:51:14.272482 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:14.272538 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 01:51:14.272622 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:14.273004 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:14.273014 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:14.273020 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 01:51:14.273163 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 01:51:14.273471 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":183,"threads_started":1}
I20260810 01:51:14.276042 6682 heartbeater.cc:347] Connected to a master server at 127.9.77.190:45181
I20260810 01:51:14.276377 6682 heartbeater.cc:464] Registering TS with master...
I20260810 01:51:14.277208 6682 heartbeater.cc:511] Master 127.9.77.190:45181 requested a full tablet report, sending...
I20260810 01:51:14.278530 6012 ts_manager.cc:194] Registered new tserver with Master: 693e202b39484507a823dd3d00bcf147 (127.9.77.131:42313)
I20260810 01:51:14.279237 6012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.77.131:39839
I20260810 01:51:14.290226 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 01:51:14.299114 6076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:14.293107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35070 (local address 127.9.77.190:45181)
0810 01:51:14.293345 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:14.293390 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:14.294086 (+ 696us) server_negotiation.cc:407] Connection header received
0810 01:51:14.294266 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:14.294269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:14.294334 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:14.294435 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:14.294783 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.295642 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.297154 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.297443 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.297537 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:14.298614 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:14.298617 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:14.298620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:14.298635 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:14.298659 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:14.298668 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:14.298786 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:14.298923 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 01:51:14.298964 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 01:51:14.303102 6012 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 01:51:14.307229 6012 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:51:14.325205 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:14.318815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40276 (local address 127.9.77.131:42313)
0810 01:51:14.319422 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:14.319437 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:14.319814 (+ 377us) server_negotiation.cc:407] Connection header received
0810 01:51:14.319927 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:14.319934 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:14.320173 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:14.320344 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:14.320813 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.321822 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.323672 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.324433 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.324564 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:14.324815 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:14.324874 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 01:51:14.324989 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":223,"threads_started":1}
I20260810 01:51:14.325209 6696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:14.318537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:37033 (local address 127.0.0.1:60528)
0810 01:51:14.319155 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:14.319216 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:14.319379 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:14.320348 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:14.320359 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:14.320389 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:14.320600 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:14.320610 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.322239 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.322243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:14.323577 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:14.323587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.324716 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.324719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:14.324830 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:14.324857 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 01:51:14.324922 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 01:51:14.325021 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":133,"thread_start_us":96,"threads_started":1}
I20260810 01:51:14.325237 6701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:14.319410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:34581 (local address 127.0.0.1:44000)
0810 01:51:14.319788 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:14.319802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:14.319891 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:14.321184 (+ 1293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:14.321187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:14.321206 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:14.321287 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:14.321292 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.322588 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.322591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:14.323762 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:14.323773 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.324825 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.324828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:14.324909 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:14.324935 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 01:51:14.324978 (+ 43us) client_negotiation.cc:240] Negotiation successful
0810 01:51:14.325045 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":74,"thread_start_us":118,"threads_started":1}
I20260810 01:51:14.325793 6697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:14.318543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:42313 (local address 127.0.0.1:40276)
0810 01:51:14.319578 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:14.319592 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:14.319697 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:14.320529 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:14.320532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:14.320547 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:14.320693 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:14.320698 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.321947 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.321953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:14.323539 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:14.323551 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.324564 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.324568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:14.324688 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:14.324730 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 01:51:14.324843 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 01:51:14.325641 (+ 798us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"mutex_wait_us":165,"thread_start_us":137,"threads_started":1}
I20260810 01:51:14.325951 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:14.319378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60528 (local address 127.9.77.129:37033)
0810 01:51:14.319691 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:14.319699 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:14.319730 (+ 31us) server_negotiation.cc:407] Connection header received
0810 01:51:14.319853 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:14.319861 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:14.320151 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:14.320764 (+ 613us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:14.320944 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.322119 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.323719 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.324556 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.325012 (+ 456us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:14.325224 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:14.325279 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 01:51:14.325759 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":108,"threads_started":1}
I20260810 01:51:14.327142 6703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:14.320146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44000 (local address 127.9.77.130:34581)
0810 01:51:14.320710 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:14.320728 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:14.320753 (+ 25us) server_negotiation.cc:407] Connection header received
0810 01:51:14.320838 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:14.320842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:14.321025 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:14.321186 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:14.321411 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.322437 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.323898 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.324705 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.325616 (+ 911us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:14.325809 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:14.326811 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0810 01:51:14.326958 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":201,"threads_started":1}
I20260810 01:51:14.329658 6201 tablet_service.cc:1467] Processing CreateTablet for tablet 8937c12d789a4137870d439866ef728f (DEFAULT_TABLE table=test-table [id=7dfe846b113f43f2a4731094168cf94e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:51:14.329660 6585 tablet_service.cc:1467] Processing CreateTablet for tablet 8937c12d789a4137870d439866ef728f (DEFAULT_TABLE table=test-table [id=7dfe846b113f43f2a4731094168cf94e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:51:14.330094 6396 tablet_service.cc:1467] Processing CreateTablet for tablet 8937c12d789a4137870d439866ef728f (DEFAULT_TABLE table=test-table [id=7dfe846b113f43f2a4731094168cf94e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:51:14.331049 6201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8937c12d789a4137870d439866ef728f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:14.331049 6585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8937c12d789a4137870d439866ef728f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:14.331156 6396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8937c12d789a4137870d439866ef728f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:14.339031 6704 tablet_bootstrap.cc:492] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428: Bootstrap starting.
I20260810 01:51:14.340317 6705 tablet_bootstrap.cc:492] T 8937c12d789a4137870d439866ef728f P b35ec97c5f9b4e6d9222dc0e2a010ba6: Bootstrap starting.
I20260810 01:51:14.342756 6705 tablet_bootstrap.cc:654] T 8937c12d789a4137870d439866ef728f P b35ec97c5f9b4e6d9222dc0e2a010ba6: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:14.342756 6704 tablet_bootstrap.cc:654] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:14.343864 6705 log.cc:824] T 8937c12d789a4137870d439866ef728f P b35ec97c5f9b4e6d9222dc0e2a010ba6: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:14.343874 6704 log.cc:824] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:14.344729 6709 tablet_bootstrap.cc:492] T 8937c12d789a4137870d439866ef728f P 693e202b39484507a823dd3d00bcf147: Bootstrap starting.
I20260810 01:51:14.346367 6704 tablet_bootstrap.cc:492] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428: No bootstrap required, opened a new log
I20260810 01:51:14.346369 6705 tablet_bootstrap.cc:492] T 8937c12d789a4137870d439866ef728f P b35ec97c5f9b4e6d9222dc0e2a010ba6: No bootstrap required, opened a new log
I20260810 01:51:14.346607 6704 ts_tablet_manager.cc:1397] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260810 01:51:14.346611 6705 ts_tablet_manager.cc:1397] T 8937c12d789a4137870d439866ef728f P b35ec97c5f9b4e6d9222dc0e2a010ba6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 01:51:14.347318 6709 tablet_bootstrap.cc:654] T 8937c12d789a4137870d439866ef728f P 693e202b39484507a823dd3d00bcf147: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:14.348361 6709 log.cc:824] T 8937c12d789a4137870d439866ef728f P 693e202b39484507a823dd3d00bcf147: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:14.350749 6709 tablet_bootstrap.cc:492] T 8937c12d789a4137870d439866ef728f P 693e202b39484507a823dd3d00bcf147: No bootstrap required, opened a new log
I20260810 01:51:14.350960 6709 ts_tablet_manager.cc:1397] T 8937c12d789a4137870d439866ef728f P 693e202b39484507a823dd3d00bcf147: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 01:51:14.350749 6704 raft_consensus.cc:348] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693e202b39484507a823dd3d00bcf147" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 42313 } } peers { permanent_uuid: "b35ec97c5f9b4e6d9222dc0e2a010ba6" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37033 } } peers { permanent_uuid: "acc648addb5b4fdcbbb6d4bf8adc7428" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 34581 } }
I20260810 01:51:14.351114 6704 raft_consensus.cc:374] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:14.351172 6704 raft_consensus.cc:729] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc648addb5b4fdcbbb6d4bf8adc7428, State: Initialized, Role: FOLLOWER
I20260810 01:51:14.351711 6704 consensus_queue.cc:260] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693e202b39484507a823dd3d00bcf147" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 42313 } } peers { permanent_uuid: "b35ec97c5f9b4e6d9222dc0e2a010ba6" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37033 } } peers { permanent_uuid: "acc648addb5b4fdcbbb6d4bf8adc7428" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 34581 } }
I20260810 01:51:14.351729 6705 raft_consensus.cc:348] T 8937c12d789a4137870d439866ef728f P b35ec97c5f9b4e6d9222dc0e2a010ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693e202b39484507a823dd3d00bcf147" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 42313 } } peers { permanent_uuid: "b35ec97c5f9b4e6d9222dc0e2a010ba6" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37033 } } peers { permanent_uuid: "acc648addb5b4fdcbbb6d4bf8adc7428" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 34581 } }
I20260810 01:51:14.352037 6705 raft_consensus.cc:374] T 8937c12d789a4137870d439866ef728f P b35ec97c5f9b4e6d9222dc0e2a010ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:14.352110 6705 raft_consensus.cc:729] T 8937c12d789a4137870d439866ef728f P b35ec97c5f9b4e6d9222dc0e2a010ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35ec97c5f9b4e6d9222dc0e2a010ba6, State: Initialized, Role: FOLLOWER
I20260810 01:51:14.352581 6705 consensus_queue.cc:260] T 8937c12d789a4137870d439866ef728f P b35ec97c5f9b4e6d9222dc0e2a010ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693e202b39484507a823dd3d00bcf147" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 42313 } } peers { permanent_uuid: "b35ec97c5f9b4e6d9222dc0e2a010ba6" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37033 } } peers { permanent_uuid: "acc648addb5b4fdcbbb6d4bf8adc7428" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 34581 } }
I20260810 01:51:14.352880 6704 ts_tablet_manager.cc:1428] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260810 01:51:14.353330 6299 heartbeater.cc:503] Master 127.9.77.190:45181 was elected leader, sending a full tablet report...
I20260810 01:51:14.353348 6490 heartbeater.cc:503] Master 127.9.77.190:45181 was elected leader, sending a full tablet report...
I20260810 01:51:14.354904 6705 ts_tablet_manager.cc:1428] T 8937c12d789a4137870d439866ef728f P b35ec97c5f9b4e6d9222dc0e2a010ba6: Time spent starting tablet: real 0.008s user 0.005s sys 0.003s
I20260810 01:51:14.356688 6709 raft_consensus.cc:348] T 8937c12d789a4137870d439866ef728f P 693e202b39484507a823dd3d00bcf147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693e202b39484507a823dd3d00bcf147" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 42313 } } peers { permanent_uuid: "b35ec97c5f9b4e6d9222dc0e2a010ba6" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37033 } } peers { permanent_uuid: "acc648addb5b4fdcbbb6d4bf8adc7428" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 34581 } }
I20260810 01:51:14.356974 6709 raft_consensus.cc:374] T 8937c12d789a4137870d439866ef728f P 693e202b39484507a823dd3d00bcf147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:14.357030 6709 raft_consensus.cc:729] T 8937c12d789a4137870d439866ef728f P 693e202b39484507a823dd3d00bcf147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 693e202b39484507a823dd3d00bcf147, State: Initialized, Role: FOLLOWER
I20260810 01:51:14.358111 6709 consensus_queue.cc:260] T 8937c12d789a4137870d439866ef728f P 693e202b39484507a823dd3d00bcf147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693e202b39484507a823dd3d00bcf147" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 42313 } } peers { permanent_uuid: "b35ec97c5f9b4e6d9222dc0e2a010ba6" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37033 } } peers { permanent_uuid: "acc648addb5b4fdcbbb6d4bf8adc7428" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 34581 } }
I20260810 01:51:14.358922 6682 heartbeater.cc:503] Master 127.9.77.190:45181 was elected leader, sending a full tablet report...
I20260810 01:51:14.359320 6709 ts_tablet_manager.cc:1428] T 8937c12d789a4137870d439866ef728f P 693e202b39484507a823dd3d00bcf147: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
W20260810 01:51:14.513468 6683 tablet.cc:2369] T 8937c12d789a4137870d439866ef728f P 693e202b39484507a823dd3d00bcf147: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 01:51:14.517030 6491 tablet.cc:2369] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 01:51:14.517086 6300 tablet.cc:2369] T 8937c12d789a4137870d439866ef728f P b35ec97c5f9b4e6d9222dc0e2a010ba6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 01:51:14.535696 6720 raft_consensus.cc:482] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:51:14.536065 6720 raft_consensus.cc:504] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693e202b39484507a823dd3d00bcf147" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 42313 } } peers { permanent_uuid: "b35ec97c5f9b4e6d9222dc0e2a010ba6" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37033 } } peers { permanent_uuid: "acc648addb5b4fdcbbb6d4bf8adc7428" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 34581 } }
I20260810 01:51:14.537647 6720 leader_election.cc:288] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 693e202b39484507a823dd3d00bcf147 (127.9.77.131:42313), b35ec97c5f9b4e6d9222dc0e2a010ba6 (127.9.77.129:37033)
I20260810 01:51:14.542148 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:14.538087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:41871 (local address 127.9.77.131:42313)
0810 01:51:14.538262 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:14.538268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:14.538440 (+ 172us) server_negotiation.cc:407] Connection header received
0810 01:51:14.538664 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:14.538666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:14.538721 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:14.538816 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:14.539186 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.539760 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.541006 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.541446 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.541584 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:14.541711 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:14.541896 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 01:51:14.541971 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 01:51:14.542181 6724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:14.537942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:42313 (local address 127.9.77.130:41871)
0810 01:51:14.538335 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:14.538349 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:14.538533 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:14.538841 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:14.538845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:14.538862 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:14.539070 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:14.539077 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.539876 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.539881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:14.540872 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:14.540881 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.541584 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.541588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:14.541688 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:14.541778 (+ 90us) client_negotiation.cc:769] Sending connection context
0810 01:51:14.541880 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 01:51:14.541968 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":132,"spinlock_wait_cycles":52480,"thread_start_us":100,"threads_started":1}
I20260810 01:51:14.542196 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:14.538151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:33245 (local address 127.9.77.129:37033)
0810 01:51:14.538359 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:14.538365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:14.538451 (+ 86us) server_negotiation.cc:407] Connection header received
0810 01:51:14.538672 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:14.538674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:14.538728 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:14.538820 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:14.539199 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.539754 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.541010 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.541437 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.541732 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:14.541791 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:14.541985 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 01:51:14.542066 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260810 01:51:14.542213 6723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:14.537987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:37033 (local address 127.9.77.130:33245)
0810 01:51:14.538335 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:14.538351 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:14.538533 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:14.538856 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:14.538860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:14.538876 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:14.539069 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:14.539077 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.539877 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.539881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:14.540870 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:14.540881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.541684 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.541690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:14.541847 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:14.541874 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 01:51:14.541967 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 01:51:14.542037 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":80,"spinlock_wait_cycles":14720,"thread_start_us":103,"threads_started":1}
I20260810 01:51:14.543023 6618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8937c12d789a4137870d439866ef728f" candidate_uuid: "acc648addb5b4fdcbbb6d4bf8adc7428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "693e202b39484507a823dd3d00bcf147" is_pre_election: true
I20260810 01:51:14.543470 6618 raft_consensus.cc:2454] T 8937c12d789a4137870d439866ef728f P 693e202b39484507a823dd3d00bcf147 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc648addb5b4fdcbbb6d4bf8adc7428 in term 0.
I20260810 01:51:14.543504 6225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8937c12d789a4137870d439866ef728f" candidate_uuid: "acc648addb5b4fdcbbb6d4bf8adc7428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35ec97c5f9b4e6d9222dc0e2a010ba6" is_pre_election: true
I20260810 01:51:14.543890 6225 raft_consensus.cc:2454] T 8937c12d789a4137870d439866ef728f P b35ec97c5f9b4e6d9222dc0e2a010ba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acc648addb5b4fdcbbb6d4bf8adc7428 in term 0.
I20260810 01:51:14.544152 6332 leader_election.cc:302] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 693e202b39484507a823dd3d00bcf147, acc648addb5b4fdcbbb6d4bf8adc7428; no voters:
I20260810 01:51:14.544510 6720 raft_consensus.cc:2781] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:51:14.544620 6720 raft_consensus.cc:482] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:51:14.544665 6720 raft_consensus.cc:3037] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:14.545929 6720 raft_consensus.cc:504] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693e202b39484507a823dd3d00bcf147" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 42313 } } peers { permanent_uuid: "b35ec97c5f9b4e6d9222dc0e2a010ba6" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37033 } } peers { permanent_uuid: "acc648addb5b4fdcbbb6d4bf8adc7428" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 34581 } }
I20260810 01:51:14.546314 6720 leader_election.cc:288] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428 [CANDIDATE]: Term 1 election: Requested vote from peers 693e202b39484507a823dd3d00bcf147 (127.9.77.131:42313), b35ec97c5f9b4e6d9222dc0e2a010ba6 (127.9.77.129:37033)
I20260810 01:51:14.546697 6225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8937c12d789a4137870d439866ef728f" candidate_uuid: "acc648addb5b4fdcbbb6d4bf8adc7428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35ec97c5f9b4e6d9222dc0e2a010ba6"
I20260810 01:51:14.546726 6618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8937c12d789a4137870d439866ef728f" candidate_uuid: "acc648addb5b4fdcbbb6d4bf8adc7428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "693e202b39484507a823dd3d00bcf147"
I20260810 01:51:14.546856 6225 raft_consensus.cc:3037] T 8937c12d789a4137870d439866ef728f P b35ec97c5f9b4e6d9222dc0e2a010ba6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:14.546886 6618 raft_consensus.cc:3037] T 8937c12d789a4137870d439866ef728f P 693e202b39484507a823dd3d00bcf147 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:14.548161 6225 raft_consensus.cc:2454] T 8937c12d789a4137870d439866ef728f P b35ec97c5f9b4e6d9222dc0e2a010ba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc648addb5b4fdcbbb6d4bf8adc7428 in term 1.
I20260810 01:51:14.548200 6618 raft_consensus.cc:2454] T 8937c12d789a4137870d439866ef728f P 693e202b39484507a823dd3d00bcf147 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acc648addb5b4fdcbbb6d4bf8adc7428 in term 1.
I20260810 01:51:14.548520 6328 leader_election.cc:302] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: acc648addb5b4fdcbbb6d4bf8adc7428, b35ec97c5f9b4e6d9222dc0e2a010ba6; no voters:
I20260810 01:51:14.548712 6720 raft_consensus.cc:2781] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:14.548985 6720 raft_consensus.cc:686] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428 [term 1 LEADER]: Becoming Leader. State: Replica: acc648addb5b4fdcbbb6d4bf8adc7428, State: Running, Role: LEADER
I20260810 01:51:14.549391 6720 consensus_queue.cc:237] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693e202b39484507a823dd3d00bcf147" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 42313 } } peers { permanent_uuid: "b35ec97c5f9b4e6d9222dc0e2a010ba6" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37033 } } peers { permanent_uuid: "acc648addb5b4fdcbbb6d4bf8adc7428" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 34581 } }
I20260810 01:51:14.553581 6012 catalog_manager.cc:5495] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428 reported cstate change: term changed from 0 to 1, leader changed from <none> to acc648addb5b4fdcbbb6d4bf8adc7428 (127.9.77.130). New cstate: current_term: 1 leader_uuid: "acc648addb5b4fdcbbb6d4bf8adc7428" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "693e202b39484507a823dd3d00bcf147" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 42313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b35ec97c5f9b4e6d9222dc0e2a010ba6" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acc648addb5b4fdcbbb6d4bf8adc7428" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 34581 } health_report { overall_health: HEALTHY } } }
I20260810 01:51:14.624693 6703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:14.620167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44008 (local address 127.9.77.130:34581)
0810 01:51:14.620354 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:14.620360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:14.620379 (+ 19us) server_negotiation.cc:407] Connection header received
0810 01:51:14.620498 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:14.620501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:14.620577 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:14.620724 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:51:14.621040 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.621630 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.622651 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:14.622920 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:14.623058 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:14.623506 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:51:14.624142 (+ 636us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:51:14.624352 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:14.624454 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 01:51:14.624502 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 01:51:14.632928 6618 raft_consensus.cc:1261] T 8937c12d789a4137870d439866ef728f P 693e202b39484507a823dd3d00bcf147 [term 1 FOLLOWER]: Refusing update from remote peer acc648addb5b4fdcbbb6d4bf8adc7428: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:51:14.632928 6225 raft_consensus.cc:1261] T 8937c12d789a4137870d439866ef728f P b35ec97c5f9b4e6d9222dc0e2a010ba6 [term 1 FOLLOWER]: Refusing update from remote peer acc648addb5b4fdcbbb6d4bf8adc7428: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:51:14.633967 6720 consensus_queue.cc:1035] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "693e202b39484507a823dd3d00bcf147" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 42313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:51:14.634284 6726 consensus_queue.cc:1035] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35ec97c5f9b4e6d9222dc0e2a010ba6" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:51:14.644963 6734 mvcc.cc:205] Tried to move back new op lower bound from 7316794059286253568 to 7316794058957283328. Current Snapshot: MvccSnapshot[applied={T|T < 7316794059286253568}]
I20260810 01:51:14.645968 6729 mvcc.cc:205] Tried to move back new op lower bound from 7316794059286253568 to 7316794058957283328. Current Snapshot: MvccSnapshot[applied={T|T < 7316794059286253568 or (T in {7316794059286253568})}]
I20260810 01:51:14.654709 6012 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:51:14.654968 6012 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:51:14.657902 6012 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7f7c9a1cd014b6db6759725a319226c: Sending DeleteTablet for 3 replicas of tablet 8937c12d789a4137870d439866ef728f
I20260810 01:51:14.659061 6585 tablet_service.cc:1514] Processing DeleteTablet for tablet 8937c12d789a4137870d439866ef728f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:51:14 UTC) from {username='slave'} at 127.0.0.1:40276
I20260810 01:51:14.659025 6201 tablet_service.cc:1514] Processing DeleteTablet for tablet 8937c12d789a4137870d439866ef728f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:51:14 UTC) from {username='slave'} at 127.0.0.1:60528
I20260810 01:51:14.659186 6396 tablet_service.cc:1514] Processing DeleteTablet for tablet 8937c12d789a4137870d439866ef728f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:51:14 UTC) from {username='slave'} at 127.0.0.1:44000
I20260810 01:51:14.659730 6743 tablet_replica.cc:330] T 8937c12d789a4137870d439866ef728f P 693e202b39484507a823dd3d00bcf147: stopping tablet replica
I20260810 01:51:14.659972 6744 tablet_replica.cc:330] T 8937c12d789a4137870d439866ef728f P b35ec97c5f9b4e6d9222dc0e2a010ba6: stopping tablet replica
I20260810 01:51:14.660254 6743 raft_consensus.cc:2229] T 8937c12d789a4137870d439866ef728f P 693e202b39484507a823dd3d00bcf147 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:51:14.660462 6744 raft_consensus.cc:2229] T 8937c12d789a4137870d439866ef728f P b35ec97c5f9b4e6d9222dc0e2a010ba6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:51:14.660563 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 6107
I20260810 01:51:14.660706 6743 raft_consensus.cc:2258] T 8937c12d789a4137870d439866ef728f P 693e202b39484507a823dd3d00bcf147 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:51:14.660944 6744 raft_consensus.cc:2258] T 8937c12d789a4137870d439866ef728f P b35ec97c5f9b4e6d9222dc0e2a010ba6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:51:14.663277 6743 ts_tablet_manager.cc:1905] T 8937c12d789a4137870d439866ef728f P 693e202b39484507a823dd3d00bcf147: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:51:14.667898 6743 ts_tablet_manager.cc:1918] T 8937c12d789a4137870d439866ef728f P 693e202b39484507a823dd3d00bcf147: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:51:14.668090 6743 log.cc:1196] T 8937c12d789a4137870d439866ef728f P 693e202b39484507a823dd3d00bcf147: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-2/wal/wals/8937c12d789a4137870d439866ef728f
I20260810 01:51:14.668504 6743 ts_tablet_manager.cc:1939] T 8937c12d789a4137870d439866ef728f P 693e202b39484507a823dd3d00bcf147: Deleting consensus metadata
I20260810 01:51:14.670363 5985 catalog_manager.cc:4841] TS 693e202b39484507a823dd3d00bcf147 (127.9.77.131:42313): tablet 8937c12d789a4137870d439866ef728f (table test-table [id=7dfe846b113f43f2a4731094168cf94e]) successfully deleted
I20260810 01:51:14.672322 6745 tablet_replica.cc:330] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428: stopping tablet replica
I20260810 01:51:14.672786 6745 raft_consensus.cc:2229] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428 [term 1 LEADER]: Raft consensus shutting down.
I20260810 01:51:14.673266 6745 raft_consensus.cc:2258] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:51:14.675089 6745 ts_tablet_manager.cc:1905] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:51:14.678112 6745 ts_tablet_manager.cc:1918] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:51:14.678259 6745 log.cc:1196] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786326392877349-9526-0/minicluster-data/ts-1/wal/wals/8937c12d789a4137870d439866ef728f
I20260810 01:51:14.678630 6745 ts_tablet_manager.cc:1939] T 8937c12d789a4137870d439866ef728f P acc648addb5b4fdcbbb6d4bf8adc7428: Deleting consensus metadata
I20260810 01:51:14.680130 5985 catalog_manager.cc:4841] TS acc648addb5b4fdcbbb6d4bf8adc7428 (127.9.77.130:34581): tablet 8937c12d789a4137870d439866ef728f (table test-table [id=7dfe846b113f43f2a4731094168cf94e]) successfully deleted
W20260810 01:51:14.681119 5982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.77.129:37033 (error 108)
I20260810 01:51:14.682300 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 6304
I20260810 01:51:14.683210 6697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:14.682071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:37033 (local address 127.0.0.1:60536)
0810 01:51:14.682998 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:14.683090 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.129:37033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":803}
W20260810 01:51:14.689728 5982 catalog_manager.cc:4584] TS b35ec97c5f9b4e6d9222dc0e2a010ba6 (127.9.77.129:37033): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8937c12d789a4137870d439866ef728f: Network error: Client connection negotiation failed: client connection to 127.9.77.129:37033: connect: Connection refused (error 111)
I20260810 01:51:14.699676 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 6493
I20260810 01:51:14.718978 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 5960
2026-08-10T01:51:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1518 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:51:14 dist-test-slave-kvfs krb5kdc[6749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:51:14 dist-test-slave-kvfs krb5kdc[6749](info): set up 2 sockets
Aug 10 01:51:14 dist-test-slave-kvfs krb5kdc[6749](info): commencing operation
krb5kdc: starting...
W20260810 01:51:16.014360 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.261s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:51:16 dist-test-slave-kvfs krb5kdc[6749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:51:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:51:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:51:16.157516 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:38143
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:35169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:38143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:16.295825 6764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:16.296190 6764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:16.296237 6764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:16.300427 6764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:16.300498 6764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:16.300518 6764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:16.300535 6764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:16.300551 6764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:16.307869 6764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:38143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:38143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:16.309367 6764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:16.310730 6764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:16.321072 6771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:16.321027 6770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:16.321219 6764 server_base.cc:1034] running on GCE node
W20260810 01:51:16.321027 6775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:16.322145 6764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:16.323540 6764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:16.324712 6764 hybrid_clock.cc:648] HybridClock initialized: now 1786326676324685 us; error 44 us; skew 500 ppm
Aug 10 01:51:16 dist-test-slave-kvfs krb5kdc[6749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326676, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:16.328473 6764 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:51:16.329695 6764 webserver.cc:460] Webserver started at http://127.9.77.190:40367/ using document root <none> and password file <none>
I20260810 01:51:16.330464 6764 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:16.330535 6764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:16.330814 6764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:16.333527 6764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "bb441c6a61814b1693600e15ba18f293"
format_stamp: "Formatted at 2026-08-10 01:51:16 on dist-test-slave-kvfs"
server_key: "6ccaba3d77feed84a0f35ed50fa4b259"
server_key_iv: "0a4c80f2c4dd1d670db552179aba3aea"
server_key_version: "encryptionkey@0"
I20260810 01:51:16.334230 6764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "bb441c6a61814b1693600e15ba18f293"
format_stamp: "Formatted at 2026-08-10 01:51:16 on dist-test-slave-kvfs"
server_key: "6ccaba3d77feed84a0f35ed50fa4b259"
server_key_iv: "0a4c80f2c4dd1d670db552179aba3aea"
server_key_version: "encryptionkey@0"
I20260810 01:51:16.339339 6764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 01:51:16.342687 6784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:16.344388 6764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:51:16.344555 6764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "bb441c6a61814b1693600e15ba18f293"
format_stamp: "Formatted at 2026-08-10 01:51:16 on dist-test-slave-kvfs"
server_key: "6ccaba3d77feed84a0f35ed50fa4b259"
server_key_iv: "0a4c80f2c4dd1d670db552179aba3aea"
server_key_version: "encryptionkey@0"
I20260810 01:51:16.344707 6764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:16.356060 6764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:16.360095 6764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:16.360374 6764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:16.370159 6764 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:38143
I20260810 01:51:16.370218 6849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:38143 every 8 connection(s)
I20260810 01:51:16.371495 6764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:51:16.375236 6850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:16.377228 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 6764
I20260810 01:51:16.377466 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:51:16.377864 9526 external_mini_cluster.cc:1428] Setting key 46e090175dd4c7ae8ad974ff258e9873
I20260810 01:51:16.383133 6850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb441c6a61814b1693600e15ba18f293: Bootstrap starting.
I20260810 01:51:16.386229 6850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb441c6a61814b1693600e15ba18f293: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:16.387331 6850 log.cc:824] T 00000000000000000000000000000000 P bb441c6a61814b1693600e15ba18f293: Log is configured to *not* fsync() on all Append() calls
Aug 10 01:51:16 dist-test-slave-kvfs krb5kdc[6749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:51:16.390350 6850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb441c6a61814b1693600e15ba18f293: No bootstrap required, opened a new log
I20260810 01:51:16.396142 6853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:16.379763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38102 (local address 127.9.77.190:38143)
0810 01:51:16.380363 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:16.380384 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:16.380434 (+ 50us) server_negotiation.cc:407] Connection header received
0810 01:51:16.381255 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:16.381279 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:16.381699 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:16.382119 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:16.383178 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:16.384433 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:16.385460 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:16.386033 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:16.386255 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:16.389437 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:16.389446 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:16.389466 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:16.389505 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:16.391894 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:16.392501 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:16.392509 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:16.392517 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:16.392691 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:16.393072 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:16.393075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:16.393077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:16.393850 (+ 773us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:16.394089 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:16.394393 (+ 304us) server_negotiation.cc:299] Negotiation successful
0810 01:51:16.394799 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":184,"threads_started":1}
I20260810 01:51:16.396231 6850 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb441c6a61814b1693600e15ba18f293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb441c6a61814b1693600e15ba18f293" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 38143 } }
I20260810 01:51:16.396574 6850 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb441c6a61814b1693600e15ba18f293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:16.396646 6850 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb441c6a61814b1693600e15ba18f293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb441c6a61814b1693600e15ba18f293, State: Initialized, Role: FOLLOWER
I20260810 01:51:16.397253 6850 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb441c6a61814b1693600e15ba18f293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb441c6a61814b1693600e15ba18f293" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 38143 } }
I20260810 01:51:16.397455 6850 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb441c6a61814b1693600e15ba18f293 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:51:16.397550 6850 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb441c6a61814b1693600e15ba18f293 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:51:16.397681 6850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb441c6a61814b1693600e15ba18f293 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:16.398988 6850 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb441c6a61814b1693600e15ba18f293 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb441c6a61814b1693600e15ba18f293" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 38143 } }
I20260810 01:51:16.399405 6850 leader_election.cc:302] T 00000000000000000000000000000000 P bb441c6a61814b1693600e15ba18f293 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb441c6a61814b1693600e15ba18f293; no voters:
I20260810 01:51:16.399830 6850 leader_election.cc:288] T 00000000000000000000000000000000 P bb441c6a61814b1693600e15ba18f293 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:51:16.399967 6855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb441c6a61814b1693600e15ba18f293 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:16.400266 6855 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb441c6a61814b1693600e15ba18f293 [term 1 LEADER]: Becoming Leader. State: Replica: bb441c6a61814b1693600e15ba18f293, State: Running, Role: LEADER
I20260810 01:51:16.400636 6855 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb441c6a61814b1693600e15ba18f293 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb441c6a61814b1693600e15ba18f293" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 38143 } }
I20260810 01:51:16.401435 6850 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb441c6a61814b1693600e15ba18f293 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:16.402750 6857 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb441c6a61814b1693600e15ba18f293 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb441c6a61814b1693600e15ba18f293" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb441c6a61814b1693600e15ba18f293" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 38143 } } }
I20260810 01:51:16.402925 6857 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb441c6a61814b1693600e15ba18f293 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:16.403304 6858 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb441c6a61814b1693600e15ba18f293 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb441c6a61814b1693600e15ba18f293. Latest consensus state: current_term: 1 leader_uuid: "bb441c6a61814b1693600e15ba18f293" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb441c6a61814b1693600e15ba18f293" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 38143 } } }
I20260810 01:51:16.403427 6858 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb441c6a61814b1693600e15ba18f293 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:16.403504 6865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:51:16.407891 6865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:51:16.414312 6865 catalog_manager.cc:1347] Generated new cluster ID: b5fb9fc77e8247eeaca2e5ab58e38cbb
I20260810 01:51:16.414397 6865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:51:16.426968 6865 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:51:16.428596 6865 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:51:16.437816 6865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb441c6a61814b1693600e15ba18f293: Generated new TSK 0
I20260810 01:51:16.439006 6865 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.77.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
Entry for principal kudu/127.9.77.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab.
I20260810 01:51:16.486936 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:0
--local_ip_for_outbound_sockets=127.9.77.129
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:38143
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:35169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:16.623172 6883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:16.623548 6883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:16.623597 6883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:16.627938 6883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:16.628033 6883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:16.628101 6883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:51:16.633810 6883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.129.keytab
--principal=kudu/127.9.77.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:38143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:16.635223 6883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:16.636650 6883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:16.648466 6891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:16.648449 6890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:16.648449 6894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:16.648953 6883 server_base.cc:1034] running on GCE node
I20260810 01:51:16.649453 6883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:16.650321 6883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:16.651535 6883 hybrid_clock.cc:648] HybridClock initialized: now 1786326676651511 us; error 31 us; skew 500 ppm
Aug 10 01:51:16 dist-test-slave-kvfs krb5kdc[6749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326676, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:16.655161 6883 init.cc:377] Logged in from keytab as kudu/127.9.77.129@KRBTEST.COM (short username kudu)
I20260810 01:51:16.656414 6883 webserver.cc:460] Webserver started at http://127.9.77.129:45563/ using document root <none> and password file <none>
I20260810 01:51:16.657182 6883 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:16.657284 6883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:16.657588 6883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:16.660306 6883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-0/data/instance:
uuid: "9f21bb0342114e388347a675ec9c6cfa"
format_stamp: "Formatted at 2026-08-10 01:51:16 on dist-test-slave-kvfs"
server_key: "7452f9b02ee263fbf5de98abd8f0028a"
server_key_iv: "47f4a692310c47a13229db3c927ec801"
server_key_version: "encryptionkey@0"
I20260810 01:51:16.661031 6883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance:
uuid: "9f21bb0342114e388347a675ec9c6cfa"
format_stamp: "Formatted at 2026-08-10 01:51:16 on dist-test-slave-kvfs"
server_key: "7452f9b02ee263fbf5de98abd8f0028a"
server_key_iv: "47f4a692310c47a13229db3c927ec801"
server_key_version: "encryptionkey@0"
I20260810 01:51:16.665902 6883 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 01:51:16.669210 6903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:16.670801 6883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:51:16.670972 6883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "9f21bb0342114e388347a675ec9c6cfa"
format_stamp: "Formatted at 2026-08-10 01:51:16 on dist-test-slave-kvfs"
server_key: "7452f9b02ee263fbf5de98abd8f0028a"
server_key_iv: "47f4a692310c47a13229db3c927ec801"
server_key_version: "encryptionkey@0"
I20260810 01:51:16.671119 6883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:16.687016 6883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:16.691190 6883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:16.691458 6883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:16.692348 6883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:51:16.693432 6883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:51:16.693508 6883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:16.693590 6883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:51:16.693635 6883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:16.705236 6883 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:39449
I20260810 01:51:16.705260 7063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:39449 every 8 connection(s)
I20260810 01:51:16.706452 6883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:51:16.706933 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 6883
I20260810 01:51:16.707111 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance
I20260810 01:51:16.707543 9526 external_mini_cluster.cc:1428] Setting key 5e78d39a04c849d1dff4b281f2da28a0
Aug 10 01:51:16 dist-test-slave-kvfs krb5kdc[6749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326676, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.129@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.77.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.130@KRBTEST.COM" created.
I20260810 01:51:16.723821 6853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:16.709170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:55323 (local address 127.9.77.190:38143)
0810 01:51:16.709353 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:16.709358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:16.710706 (+ 1348us) server_negotiation.cc:407] Connection header received
0810 01:51:16.712346 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:16.712350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:16.712416 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:16.712533 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:16.713960 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:16.714593 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:16.715883 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:16.716198 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:16.716321 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:16.719599 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:16.719602 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:16.719604 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:16.719634 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:16.721284 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:16.721915 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:16.721919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:16.721921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:16.722014 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:16.722392 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:16.722395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:16.722397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:16.722614 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:16.722725 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:16.723438 (+ 713us) server_negotiation.cc:299] Negotiation successful
0810 01:51:16.723604 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 01:51:16.724916 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:16.709853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:38143 (local address 127.9.77.129:55323)
0810 01:51:16.710530 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:16.710584 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:16.712116 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:16.712750 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:16.712758 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:16.713188 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:16.713796 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:16.713811 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:16.714730 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:16.714733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:16.715740 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:16.715749 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:16.716382 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:16.716387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:16.716588 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:16.717426 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:51:16.717453 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:16.719395 (+ 1942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:16.721416 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:16.721423 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:16.721444 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:16.721806 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:16.722109 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:16.722113 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:16.722115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:16.722278 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:16.722723 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:16.722732 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:16.723118 (+ 386us) client_negotiation.cc:769] Sending connection context
0810 01:51:16.723356 (+ 238us) client_negotiation.cc:240] Negotiation successful
0810 01:51:16.723773 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":191,"threads_started":1}
I20260810 01:51:16.726516 7065 heartbeater.cc:347] Connected to a master server at 127.9.77.190:38143
I20260810 01:51:16.726861 7065 heartbeater.cc:464] Registering TS with master...
I20260810 01:51:16.727662 7065 heartbeater.cc:511] Master 127.9.77.190:38143 requested a full tablet report, sending...
I20260810 01:51:16.729822 6802 ts_manager.cc:194] Registered new tserver with Master: 9f21bb0342114e388347a675ec9c6cfa (127.9.77.129:39449)
I20260810 01:51:16.731868 6802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.129@KRBTEST.COM'} at 127.9.77.129:55323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
Entry for principal kudu/127.9.77.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab.
I20260810 01:51:16.745036 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:0
--local_ip_for_outbound_sockets=127.9.77.130
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:38143
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:35169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:16.878005 7076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:16.878327 7076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:16.878371 7076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:16.882474 7076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:16.882540 7076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:16.882597 7076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:51:16.888098 7076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.130.keytab
--principal=kudu/127.9.77.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:38143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:16.889504 7076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:16.890821 7076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:16.902859 7083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:16.902863 7086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:16.902922 7084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:16.903160 7076 server_base.cc:1034] running on GCE node
I20260810 01:51:16.903725 7076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:16.904614 7076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:16.905819 7076 hybrid_clock.cc:648] HybridClock initialized: now 1786326676905794 us; error 34 us; skew 500 ppm
Aug 10 01:51:16 dist-test-slave-kvfs krb5kdc[6749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326676, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:16.914546 7076 init.cc:377] Logged in from keytab as kudu/127.9.77.130@KRBTEST.COM (short username kudu)
I20260810 01:51:16.915728 7076 webserver.cc:460] Webserver started at http://127.9.77.130:34645/ using document root <none> and password file <none>
I20260810 01:51:16.916550 7076 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:16.916654 7076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:16.916952 7076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:16.919560 7076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-1/data/instance:
uuid: "031521c20ae64fcb98088a9f7f678f57"
format_stamp: "Formatted at 2026-08-10 01:51:16 on dist-test-slave-kvfs"
server_key: "067a0ea0508a4f6dc30d214c2e8b05f8"
server_key_iv: "02010f1c26ad572449bdae6383854772"
server_key_version: "encryptionkey@0"
I20260810 01:51:16.920302 7076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance:
uuid: "031521c20ae64fcb98088a9f7f678f57"
format_stamp: "Formatted at 2026-08-10 01:51:16 on dist-test-slave-kvfs"
server_key: "067a0ea0508a4f6dc30d214c2e8b05f8"
server_key_iv: "02010f1c26ad572449bdae6383854772"
server_key_version: "encryptionkey@0"
I20260810 01:51:16.924952 7076 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 01:51:16.927894 7100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:16.929481 7076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:51:16.929646 7076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "031521c20ae64fcb98088a9f7f678f57"
format_stamp: "Formatted at 2026-08-10 01:51:16 on dist-test-slave-kvfs"
server_key: "067a0ea0508a4f6dc30d214c2e8b05f8"
server_key_iv: "02010f1c26ad572449bdae6383854772"
server_key_version: "encryptionkey@0"
I20260810 01:51:16.929787 7076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:16.983116 7076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:16.987059 7076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:16.987315 7076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:16.988158 7076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:51:16.989202 7076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:51:16.989279 7076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:16.989364 7076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:51:16.989416 7076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:16.999677 7076 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:37121
I20260810 01:51:16.999707 7267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:37121 every 8 connection(s)
I20260810 01:51:17.000869 7076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:51:17.006127 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 7076
I20260810 01:51:17.006309 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance
I20260810 01:51:17.006690 9526 external_mini_cluster.cc:1428] Setting key 2c50248a7aa06547e9270b6604a12fd2
Aug 10 01:51:17 dist-test-slave-kvfs krb5kdc[6749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326676, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.130@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:51:17.017702 6853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:17.003242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:54689 (local address 127.9.77.190:38143)
0810 01:51:17.003430 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:17.003436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:17.004605 (+ 1169us) server_negotiation.cc:407] Connection header received
0810 01:51:17.006188 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:17.006191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:17.006266 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:17.006395 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:17.008128 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.008940 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.010214 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.010621 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.010749 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:17.013653 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:17.013657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:17.013659 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:17.013685 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:17.015313 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:17.015891 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:17.015894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:17.015897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:17.015988 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:17.016406 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:17.016409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:17.016411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:17.016625 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:17.016739 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:17.017329 (+ 590us) server_negotiation.cc:299] Negotiation successful
0810 01:51:17.017497 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 01:51:17.018743 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:17.003796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:38143 (local address 127.9.77.130:54689)
0810 01:51:17.004432 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:17.004483 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:17.005936 (+ 1453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:17.006605 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:17.006614 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:17.007129 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:17.007925 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:17.007948 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.009079 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.009082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:17.010080 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:17.010090 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.010789 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.010791 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:17.010973 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:17.011708 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:51:17.011726 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:17.013465 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:17.015460 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:17.015467 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:17.015486 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:17.015776 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:17.016129 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:17.016132 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:17.016134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:17.016289 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:17.016737 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:17.016745 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:17.017057 (+ 312us) client_negotiation.cc:769] Sending connection context
0810 01:51:17.017272 (+ 215us) client_negotiation.cc:240] Negotiation successful
0810 01:51:17.017660 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":194,"threads_started":1}
WARNING: no policy specified for kudu/127.9.77.131@KRBTEST.COM; defaulting to no policy
I20260810 01:51:17.020309 7268 heartbeater.cc:347] Connected to a master server at 127.9.77.190:38143
I20260810 01:51:17.020623 7268 heartbeater.cc:464] Registering TS with master...
I20260810 01:51:17.021252 7268 heartbeater.cc:511] Master 127.9.77.190:38143 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.131@KRBTEST.COM" created.
I20260810 01:51:17.022732 6802 ts_manager.cc:194] Registered new tserver with Master: 031521c20ae64fcb98088a9f7f678f57 (127.9.77.130:37121)
I20260810 01:51:17.023428 6802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.130@KRBTEST.COM'} at 127.9.77.130:54689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
Entry for principal kudu/127.9.77.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab.
I20260810 01:51:17.041399 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:0
--local_ip_for_outbound_sockets=127.9.77.131
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:38143
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:35169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:17.175912 7274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:17.176267 7274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:17.176347 7274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:17.180493 7274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:17.180562 7274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:17.180625 7274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:51:17.187036 7274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.131.keytab
--principal=kudu/127.9.77.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:38143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:17.188450 7274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:17.189700 7274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:17.201609 7286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:17.201562 7280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:17.201581 7281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:17.201813 7274 server_base.cc:1034] running on GCE node
I20260810 01:51:17.202555 7274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:17.203471 7274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:17.204643 7274 hybrid_clock.cc:648] HybridClock initialized: now 1786326677204626 us; error 49 us; skew 500 ppm
Aug 10 01:51:17 dist-test-slave-kvfs krb5kdc[6749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326677, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:17.207986 7274 init.cc:377] Logged in from keytab as kudu/127.9.77.131@KRBTEST.COM (short username kudu)
I20260810 01:51:17.209177 7274 webserver.cc:460] Webserver started at http://127.9.77.131:38179/ using document root <none> and password file <none>
I20260810 01:51:17.209900 7274 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:17.210003 7274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:17.210292 7274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:17.212954 7274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-2/data/instance:
uuid: "94fca0f2ef064318a6123861fb408f75"
format_stamp: "Formatted at 2026-08-10 01:51:17 on dist-test-slave-kvfs"
server_key: "8c9e389fef7fcbf7049ff3207f6b3edc"
server_key_iv: "3f899f1aa0660ea20918b82a39f80790"
server_key_version: "encryptionkey@0"
I20260810 01:51:17.213631 7274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance:
uuid: "94fca0f2ef064318a6123861fb408f75"
format_stamp: "Formatted at 2026-08-10 01:51:17 on dist-test-slave-kvfs"
server_key: "8c9e389fef7fcbf7049ff3207f6b3edc"
server_key_iv: "3f899f1aa0660ea20918b82a39f80790"
server_key_version: "encryptionkey@0"
I20260810 01:51:17.218370 7274 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 01:51:17.221490 7297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:17.223048 7274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 01:51:17.223233 7274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "94fca0f2ef064318a6123861fb408f75"
format_stamp: "Formatted at 2026-08-10 01:51:17 on dist-test-slave-kvfs"
server_key: "8c9e389fef7fcbf7049ff3207f6b3edc"
server_key_iv: "3f899f1aa0660ea20918b82a39f80790"
server_key_version: "encryptionkey@0"
I20260810 01:51:17.223383 7274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:17.238451 7274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:17.242823 7274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:17.243089 7274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:17.243881 7274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:51:17.245057 7274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:51:17.245121 7274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:17.245195 7274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:51:17.245229 7274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:17.256848 7274 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:35505
I20260810 01:51:17.256937 7478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:35505 every 8 connection(s)
I20260810 01:51:17.258008 7274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:51:17.260718 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 7274
I20260810 01:51:17.260898 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance
I20260810 01:51:17.261250 9526 external_mini_cluster.cc:1428] Setting key a6b412b5c555e1dd2eb5d90a554114f6
Aug 10 01:51:17 dist-test-slave-kvfs krb5kdc[6749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326677, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.131@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:51:17.274259 6853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:17.260444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:39637 (local address 127.9.77.190:38143)
0810 01:51:17.260653 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:17.260659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:17.261781 (+ 1122us) server_negotiation.cc:407] Connection header received
0810 01:51:17.263763 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:17.263767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:17.263831 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:17.263918 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:17.265383 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.265987 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.267244 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.267510 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.267622 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:17.270601 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:17.270605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:17.270607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:17.270634 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:17.271992 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:17.272580 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:17.272584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:17.272586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:17.272677 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:17.273044 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:17.273047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:17.273049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:17.273258 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:17.273382 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:17.273929 (+ 547us) server_negotiation.cc:299] Negotiation successful
0810 01:51:17.274068 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 01:51:17.275514 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:17.260984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:38143 (local address 127.9.77.131:39637)
0810 01:51:17.261592 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:17.261657 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:17.263533 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:17.264202 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:17.264211 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:17.264633 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:17.265229 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:17.265244 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.266118 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.266122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:17.267112 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:17.267122 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.267677 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.267681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:17.267891 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:17.268758 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:51:17.268776 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:17.270420 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:17.272160 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:17.272166 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:17.272182 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:17.272471 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:17.272775 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:17.272779 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:17.272781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:17.272942 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:17.273369 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:17.273377 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:17.273664 (+ 287us) client_negotiation.cc:769] Sending connection context
0810 01:51:17.273893 (+ 229us) client_negotiation.cc:240] Negotiation successful
0810 01:51:17.274264 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":143,"threads_started":1}
I20260810 01:51:17.276899 7479 heartbeater.cc:347] Connected to a master server at 127.9.77.190:38143
I20260810 01:51:17.277195 7479 heartbeater.cc:464] Registering TS with master...
I20260810 01:51:17.277806 7479 heartbeater.cc:511] Master 127.9.77.190:38143 requested a full tablet report, sending...
I20260810 01:51:17.278971 6802 ts_manager.cc:194] Registered new tserver with Master: 94fca0f2ef064318a6123861fb408f75 (127.9.77.131:35505)
I20260810 01:51:17.279640 6802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.77.131@KRBTEST.COM'} at 127.9.77.131:39637
I20260810 01:51:17.287957 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 01:51:17.298375 6853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:17.290439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38116 (local address 127.9.77.190:38143)
0810 01:51:17.290598 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:17.290602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:17.290795 (+ 193us) server_negotiation.cc:407] Connection header received
0810 01:51:17.290917 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:17.290919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:17.290964 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:17.291035 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:17.291816 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.292431 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.293336 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.293553 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.293644 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:17.295120 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:17.295123 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:17.295125 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:17.295148 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:17.296695 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:17.297118 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:17.297120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:17.297121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:17.297199 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:17.297443 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:17.297445 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:17.297446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:17.297693 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:17.297808 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:17.298026 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 01:51:17.298129 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 01:51:17.301712 6802 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 01:51:17.305675 6802 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:51:17.324697 7494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:17.317033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:35505 (local address 127.0.0.1:42798)
0810 01:51:17.319091 (+ 2058us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:17.319106 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:17.319209 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:17.320100 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:17.320104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:17.320123 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:17.320249 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:17.320259 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.321463 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.321467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:17.322540 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:17.322552 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.323952 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.323955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:17.324096 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:17.324201 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 01:51:17.324347 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 01:51:17.324503 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1978,"mutex_wait_us":71,"thread_start_us":126,"threads_started":1}
I20260810 01:51:17.324710 7495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:17.317894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42798 (local address 127.9.77.131:35505)
0810 01:51:17.318304 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:17.318324 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:17.319342 (+ 1018us) server_negotiation.cc:407] Connection header received
0810 01:51:17.319447 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:17.319452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:17.319680 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:17.319842 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:17.320395 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.321325 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.322929 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.323803 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.323942 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:17.324141 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:17.324389 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 01:51:17.324506 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":185,"threads_started":1}
I20260810 01:51:17.325451 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:17.316996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:37121 (local address 127.0.0.1:50904)
0810 01:51:17.317387 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:17.317454 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:17.317603 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:17.318988 (+ 1385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:17.318998 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:17.319034 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:17.319253 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:17.319265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.320907 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.320913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:17.322552 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:17.322562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.324101 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.324104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:17.324290 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:17.324315 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 01:51:17.324389 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 01:51:17.325307 (+ 918us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":851,"thread_start_us":120,"threads_started":1}
I20260810 01:51:17.325621 7496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:17.317945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50904 (local address 127.9.77.130:37121)
0810 01:51:17.318343 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:17.318358 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:17.318403 (+ 45us) server_negotiation.cc:407] Connection header received
0810 01:51:17.318500 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:17.318505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:17.318766 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:17.318986 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:17.319448 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.320742 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.322750 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.323702 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.323841 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:17.324067 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:17.324487 (+ 420us) server_negotiation.cc:299] Negotiation successful
0810 01:51:17.324595 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":181,"threads_started":1}
I20260810 01:51:17.325645 7500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:17.319648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52096 (local address 127.9.77.129:39449)
0810 01:51:17.320105 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:17.320111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:17.320317 (+ 206us) server_negotiation.cc:407] Connection header received
0810 01:51:17.320448 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:17.320453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:17.320629 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:17.320785 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:17.321028 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.322174 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.323828 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.324808 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.324933 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:17.325071 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:17.325384 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 01:51:17.325491 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":84,"threads_started":1}
I20260810 01:51:17.325487 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:17.319116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:39449 (local address 127.0.0.1:52096)
0810 01:51:17.320192 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:17.320208 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:17.320315 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:17.320781 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:17.320784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:17.320810 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:17.320899 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:17.320905 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.322301 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.322304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:17.323680 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:17.323692 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.325130 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.325134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:17.325223 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:17.325246 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 01:51:17.325322 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 01:51:17.325382 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"thread_start_us":121,"threads_started":1}
I20260810 01:51:17.328850 7172 tablet_service.cc:1467] Processing CreateTablet for tablet ca3a5948333f47638b1353db4705a32f (DEFAULT_TABLE table=test-table [id=768f95c393b942a0ac64156dc9f27fb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:51:17.328853 7376 tablet_service.cc:1467] Processing CreateTablet for tablet ca3a5948333f47638b1353db4705a32f (DEFAULT_TABLE table=test-table [id=768f95c393b942a0ac64156dc9f27fb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:51:17.330027 7172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca3a5948333f47638b1353db4705a32f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:17.330044 7376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca3a5948333f47638b1353db4705a32f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:17.330825 6969 tablet_service.cc:1467] Processing CreateTablet for tablet ca3a5948333f47638b1353db4705a32f (DEFAULT_TABLE table=test-table [id=768f95c393b942a0ac64156dc9f27fb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:51:17.331890 6969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca3a5948333f47638b1353db4705a32f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:17.338551 7501 tablet_bootstrap.cc:492] T ca3a5948333f47638b1353db4705a32f P 031521c20ae64fcb98088a9f7f678f57: Bootstrap starting.
I20260810 01:51:17.340201 7504 tablet_bootstrap.cc:492] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa: Bootstrap starting.
I20260810 01:51:17.340854 7503 tablet_bootstrap.cc:492] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75: Bootstrap starting.
I20260810 01:51:17.341262 7501 tablet_bootstrap.cc:654] T ca3a5948333f47638b1353db4705a32f P 031521c20ae64fcb98088a9f7f678f57: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:17.342549 7501 log.cc:824] T ca3a5948333f47638b1353db4705a32f P 031521c20ae64fcb98088a9f7f678f57: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:17.342849 7504 tablet_bootstrap.cc:654] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:17.343118 7503 tablet_bootstrap.cc:654] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:17.343868 7504 log.cc:824] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:17.344547 7503 log.cc:824] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:17.346441 7501 tablet_bootstrap.cc:492] T ca3a5948333f47638b1353db4705a32f P 031521c20ae64fcb98088a9f7f678f57: No bootstrap required, opened a new log
I20260810 01:51:17.346457 7503 tablet_bootstrap.cc:492] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75: No bootstrap required, opened a new log
I20260810 01:51:17.346457 7504 tablet_bootstrap.cc:492] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa: No bootstrap required, opened a new log
I20260810 01:51:17.346673 7501 ts_tablet_manager.cc:1397] T ca3a5948333f47638b1353db4705a32f P 031521c20ae64fcb98088a9f7f678f57: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260810 01:51:17.346673 7504 ts_tablet_manager.cc:1397] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 01:51:17.346673 7503 ts_tablet_manager.cc:1397] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 01:51:17.350464 7501 raft_consensus.cc:348] T ca3a5948333f47638b1353db4705a32f P 031521c20ae64fcb98088a9f7f678f57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031521c20ae64fcb98088a9f7f678f57" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37121 } } peers { permanent_uuid: "94fca0f2ef064318a6123861fb408f75" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35505 } } peers { permanent_uuid: "9f21bb0342114e388347a675ec9c6cfa" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39449 } }
I20260810 01:51:17.350764 7501 raft_consensus.cc:374] T ca3a5948333f47638b1353db4705a32f P 031521c20ae64fcb98088a9f7f678f57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:17.350831 7501 raft_consensus.cc:729] T ca3a5948333f47638b1353db4705a32f P 031521c20ae64fcb98088a9f7f678f57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031521c20ae64fcb98088a9f7f678f57, State: Initialized, Role: FOLLOWER
I20260810 01:51:17.351027 7503 raft_consensus.cc:348] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031521c20ae64fcb98088a9f7f678f57" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37121 } } peers { permanent_uuid: "94fca0f2ef064318a6123861fb408f75" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35505 } } peers { permanent_uuid: "9f21bb0342114e388347a675ec9c6cfa" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39449 } }
I20260810 01:51:17.351318 7503 raft_consensus.cc:374] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:17.351383 7503 raft_consensus.cc:729] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94fca0f2ef064318a6123861fb408f75, State: Initialized, Role: FOLLOWER
I20260810 01:51:17.351373 7501 consensus_queue.cc:260] T ca3a5948333f47638b1353db4705a32f P 031521c20ae64fcb98088a9f7f678f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031521c20ae64fcb98088a9f7f678f57" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37121 } } peers { permanent_uuid: "94fca0f2ef064318a6123861fb408f75" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35505 } } peers { permanent_uuid: "9f21bb0342114e388347a675ec9c6cfa" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39449 } }
I20260810 01:51:17.351639 7504 raft_consensus.cc:348] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031521c20ae64fcb98088a9f7f678f57" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37121 } } peers { permanent_uuid: "94fca0f2ef064318a6123861fb408f75" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35505 } } peers { permanent_uuid: "9f21bb0342114e388347a675ec9c6cfa" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39449 } }
I20260810 01:51:17.351897 7504 raft_consensus.cc:374] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:17.351961 7504 raft_consensus.cc:729] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f21bb0342114e388347a675ec9c6cfa, State: Initialized, Role: FOLLOWER
I20260810 01:51:17.351912 7503 consensus_queue.cc:260] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031521c20ae64fcb98088a9f7f678f57" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37121 } } peers { permanent_uuid: "94fca0f2ef064318a6123861fb408f75" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35505 } } peers { permanent_uuid: "9f21bb0342114e388347a675ec9c6cfa" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39449 } }
I20260810 01:51:17.352398 7268 heartbeater.cc:503] Master 127.9.77.190:38143 was elected leader, sending a full tablet report...
I20260810 01:51:17.352509 7504 consensus_queue.cc:260] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031521c20ae64fcb98088a9f7f678f57" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37121 } } peers { permanent_uuid: "94fca0f2ef064318a6123861fb408f75" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35505 } } peers { permanent_uuid: "9f21bb0342114e388347a675ec9c6cfa" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39449 } }
I20260810 01:51:17.352653 7501 ts_tablet_manager.cc:1428] T ca3a5948333f47638b1353db4705a32f P 031521c20ae64fcb98088a9f7f678f57: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 01:51:17.352977 7503 ts_tablet_manager.cc:1428] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 01:51:17.353454 7504 ts_tablet_manager.cc:1428] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20260810 01:51:17.353461 7479 heartbeater.cc:503] Master 127.9.77.190:38143 was elected leader, sending a full tablet report...
I20260810 01:51:17.353780 7065 heartbeater.cc:503] Master 127.9.77.190:38143 was elected leader, sending a full tablet report...
I20260810 01:51:17.444165 7511 raft_consensus.cc:482] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:51:17.444517 7511 raft_consensus.cc:504] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031521c20ae64fcb98088a9f7f678f57" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37121 } } peers { permanent_uuid: "94fca0f2ef064318a6123861fb408f75" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35505 } } peers { permanent_uuid: "9f21bb0342114e388347a675ec9c6cfa" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39449 } }
I20260810 01:51:17.445468 7510 raft_consensus.cc:482] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:51:17.445699 7510 raft_consensus.cc:504] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031521c20ae64fcb98088a9f7f678f57" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37121 } } peers { permanent_uuid: "94fca0f2ef064318a6123861fb408f75" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35505 } } peers { permanent_uuid: "9f21bb0342114e388347a675ec9c6cfa" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39449 } }
I20260810 01:51:17.446303 7511 leader_election.cc:288] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 031521c20ae64fcb98088a9f7f678f57 (127.9.77.130:37121), 94fca0f2ef064318a6123861fb408f75 (127.9.77.131:35505)
I20260810 01:51:17.446991 7510 leader_election.cc:288] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 031521c20ae64fcb98088a9f7f678f57 (127.9.77.130:37121), 9f21bb0342114e388347a675ec9c6cfa (127.9.77.129:39449)
I20260810 01:51:17.452600 7495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:17.446872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:47925 (local address 127.9.77.131:35505)
0810 01:51:17.447113 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:17.447120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:17.447684 (+ 564us) server_negotiation.cc:407] Connection header received
0810 01:51:17.447929 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:17.447934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:17.447999 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:17.448071 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:17.448565 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.449566 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.451250 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.451822 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.451915 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:17.451979 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:17.452354 (+ 375us) server_negotiation.cc:299] Negotiation successful
0810 01:51:17.452435 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"spinlock_wait_cycles":2560}
I20260810 01:51:17.452757 7500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:17.447701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:51475 (local address 127.9.77.129:39449)
0810 01:51:17.447798 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:17.447803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:17.447921 (+ 118us) server_negotiation.cc:407] Connection header received
0810 01:51:17.448142 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:17.448145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:17.448190 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:17.448240 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:17.448698 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.449415 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.451223 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.451822 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.451915 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:17.451979 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:17.452530 (+ 551us) server_negotiation.cc:299] Negotiation successful
0810 01:51:17.452601 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45,"spinlock_wait_cycles":49920}
I20260810 01:51:17.452607 7514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:17.446604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:35505 (local address 127.9.77.129:47925)
0810 01:51:17.447599 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:17.447612 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:17.447781 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:17.448240 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:17.448243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:17.448255 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:17.448430 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:17.448439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.449689 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.449694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:17.451120 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:17.451133 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.452103 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.452106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:17.452217 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:17.452235 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 01:51:17.452313 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 01:51:17.452384 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"mutex_wait_us":133,"spinlock_wait_cycles":6272,"thread_start_us":509,"threads_started":1}
I20260810 01:51:17.453058 7517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:17.447316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:39449 (local address 127.9.77.131:51475)
0810 01:51:17.447840 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:17.447856 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:17.447988 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:17.448354 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:17.448357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:17.448372 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:17.448512 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:17.448522 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.449554 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.449557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:17.451079 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:17.451091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.452103 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.452106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:17.452211 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:17.452232 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 01:51:17.452313 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 01:51:17.452384 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":60,"spinlock_wait_cycles":10240,"thread_start_us":123,"threads_started":1}
I20260810 01:51:17.453264 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:17.447132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:37121 (local address 127.9.77.131:50687)
0810 01:51:17.447323 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:17.447345 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:17.447488 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:17.448202 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:17.448208 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:17.448232 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:17.448378 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:17.448387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.450321 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.450326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:17.451570 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:17.451580 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.452884 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.452887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:17.452996 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:17.453028 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 01:51:17.453080 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 01:51:17.453153 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":1}
I20260810 01:51:17.453337 7496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:17.446854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:34739 (local address 127.9.77.130:37121)
0810 01:51:17.447052 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:17.447058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:17.447659 (+ 601us) server_negotiation.cc:407] Connection header received
0810 01:51:17.447947 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:17.447951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:17.448011 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:17.448106 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:17.448707 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.449421 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.451553 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.452237 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.452383 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:17.452435 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:17.453109 (+ 674us) server_negotiation.cc:299] Negotiation successful
0810 01:51:17.453192 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":72}
I20260810 01:51:17.453763 7515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:17.447204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:50687 (local address 127.9.77.130:37121)
0810 01:51:17.447947 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:17.447961 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:17.447972 (+ 11us) server_negotiation.cc:407] Connection header received
0810 01:51:17.448005 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:17.448008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:17.448066 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:17.448506 (+ 440us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:17.448654 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.450171 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.451693 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.452735 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.453453 (+ 718us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:17.453504 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:17.453540 (+ 36us) server_negotiation.cc:299] Negotiation successful
0810 01:51:17.453596 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":680,"thread_start_us":129,"threads_started":1}
I20260810 01:51:17.453961 6996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca3a5948333f47638b1353db4705a32f" candidate_uuid: "94fca0f2ef064318a6123861fb408f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f21bb0342114e388347a675ec9c6cfa" is_pre_election: true
I20260810 01:51:17.453967 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca3a5948333f47638b1353db4705a32f" candidate_uuid: "9f21bb0342114e388347a675ec9c6cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94fca0f2ef064318a6123861fb408f75" is_pre_election: true
I20260810 01:51:17.454178 7512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:17.446600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:37121 (local address 127.9.77.129:34739)
0810 01:51:17.447556 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:17.447572 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:17.447780 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:17.448272 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:17.448275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:17.448452 (+ 177us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:17.448581 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:17.448593 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.449561 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.449564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:17.451408 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:17.451419 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.452841 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.452848 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:17.452957 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:17.452982 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 01:51:17.453926 (+ 944us) client_negotiation.cc:240] Negotiation successful
0810 01:51:17.454021 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"mutex_wait_us":155,"spinlock_wait_cycles":44672,"thread_start_us":273,"threads_started":1}
I20260810 01:51:17.454245 6996 raft_consensus.cc:2454] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94fca0f2ef064318a6123861fb408f75 in term 0.
I20260810 01:51:17.454243 7404 raft_consensus.cc:2454] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f21bb0342114e388347a675ec9c6cfa in term 0.
I20260810 01:51:17.454777 7300 leader_election.cc:302] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94fca0f2ef064318a6123861fb408f75, 9f21bb0342114e388347a675ec9c6cfa; no voters:
I20260810 01:51:17.454792 6906 leader_election.cc:302] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94fca0f2ef064318a6123861fb408f75, 9f21bb0342114e388347a675ec9c6cfa; no voters:
I20260810 01:51:17.455129 7510 raft_consensus.cc:2781] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:51:17.455090 7199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca3a5948333f47638b1353db4705a32f" candidate_uuid: "9f21bb0342114e388347a675ec9c6cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031521c20ae64fcb98088a9f7f678f57" is_pre_election: true
I20260810 01:51:17.455225 7510 raft_consensus.cc:482] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:51:17.455173 7511 raft_consensus.cc:2781] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:51:17.455281 7510 raft_consensus.cc:3037] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:17.455284 7511 raft_consensus.cc:482] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:51:17.455344 7511 raft_consensus.cc:3037] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:17.455448 7199 raft_consensus.cc:2454] T ca3a5948333f47638b1353db4705a32f P 031521c20ae64fcb98088a9f7f678f57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f21bb0342114e388347a675ec9c6cfa in term 0.
I20260810 01:51:17.456772 7510 raft_consensus.cc:504] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031521c20ae64fcb98088a9f7f678f57" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37121 } } peers { permanent_uuid: "94fca0f2ef064318a6123861fb408f75" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35505 } } peers { permanent_uuid: "9f21bb0342114e388347a675ec9c6cfa" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39449 } }
I20260810 01:51:17.456780 7511 raft_consensus.cc:504] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031521c20ae64fcb98088a9f7f678f57" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37121 } } peers { permanent_uuid: "94fca0f2ef064318a6123861fb408f75" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35505 } } peers { permanent_uuid: "9f21bb0342114e388347a675ec9c6cfa" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39449 } }
I20260810 01:51:17.457185 7510 leader_election.cc:288] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75 [CANDIDATE]: Term 1 election: Requested vote from peers 031521c20ae64fcb98088a9f7f678f57 (127.9.77.130:37121), 9f21bb0342114e388347a675ec9c6cfa (127.9.77.129:39449)
I20260810 01:51:17.457195 7511 leader_election.cc:288] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa [CANDIDATE]: Term 1 election: Requested vote from peers 031521c20ae64fcb98088a9f7f678f57 (127.9.77.130:37121), 94fca0f2ef064318a6123861fb408f75 (127.9.77.131:35505)
I20260810 01:51:17.455087 7198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca3a5948333f47638b1353db4705a32f" candidate_uuid: "94fca0f2ef064318a6123861fb408f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031521c20ae64fcb98088a9f7f678f57" is_pre_election: true
I20260810 01:51:17.457446 7198 raft_consensus.cc:2454] T ca3a5948333f47638b1353db4705a32f P 031521c20ae64fcb98088a9f7f678f57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94fca0f2ef064318a6123861fb408f75 in term 0.
I20260810 01:51:17.457696 7199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca3a5948333f47638b1353db4705a32f" candidate_uuid: "9f21bb0342114e388347a675ec9c6cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031521c20ae64fcb98088a9f7f678f57"
I20260810 01:51:17.457721 6996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca3a5948333f47638b1353db4705a32f" candidate_uuid: "94fca0f2ef064318a6123861fb408f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f21bb0342114e388347a675ec9c6cfa"
I20260810 01:51:17.457772 7197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca3a5948333f47638b1353db4705a32f" candidate_uuid: "94fca0f2ef064318a6123861fb408f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031521c20ae64fcb98088a9f7f678f57"
I20260810 01:51:17.457818 7199 raft_consensus.cc:3037] T ca3a5948333f47638b1353db4705a32f P 031521c20ae64fcb98088a9f7f678f57 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:17.457893 6996 raft_consensus.cc:2379] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 94fca0f2ef064318a6123861fb408f75 in current term 1: Already voted for candidate 9f21bb0342114e388347a675ec9c6cfa in this term.
W20260810 01:51:17.458196 7066 tablet.cc:2369] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 01:51:17.458678 7298 leader_election.cc:302] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 94fca0f2ef064318a6123861fb408f75; no voters: 031521c20ae64fcb98088a9f7f678f57, 9f21bb0342114e388347a675ec9c6cfa
I20260810 01:51:17.458724 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca3a5948333f47638b1353db4705a32f" candidate_uuid: "9f21bb0342114e388347a675ec9c6cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94fca0f2ef064318a6123861fb408f75"
I20260810 01:51:17.458885 7404 raft_consensus.cc:2379] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f21bb0342114e388347a675ec9c6cfa in current term 1: Already voted for candidate 94fca0f2ef064318a6123861fb408f75 in this term.
I20260810 01:51:17.458990 7510 raft_consensus.cc:2726] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 01:51:17.459408 7199 raft_consensus.cc:2454] T ca3a5948333f47638b1353db4705a32f P 031521c20ae64fcb98088a9f7f678f57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f21bb0342114e388347a675ec9c6cfa in term 1.
I20260810 01:51:17.459671 6904 leader_election.cc:302] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 031521c20ae64fcb98088a9f7f678f57, 9f21bb0342114e388347a675ec9c6cfa; no voters: 94fca0f2ef064318a6123861fb408f75
I20260810 01:51:17.459901 7511 raft_consensus.cc:2781] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:17.460208 7511 raft_consensus.cc:686] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa [term 1 LEADER]: Becoming Leader. State: Replica: 9f21bb0342114e388347a675ec9c6cfa, State: Running, Role: LEADER
I20260810 01:51:17.460610 7511 consensus_queue.cc:237] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031521c20ae64fcb98088a9f7f678f57" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37121 } } peers { permanent_uuid: "94fca0f2ef064318a6123861fb408f75" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35505 } } peers { permanent_uuid: "9f21bb0342114e388347a675ec9c6cfa" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39449 } }
I20260810 01:51:17.464761 6800 catalog_manager.cc:5495] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f21bb0342114e388347a675ec9c6cfa (127.9.77.129). New cstate: current_term: 1 leader_uuid: "9f21bb0342114e388347a675ec9c6cfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031521c20ae64fcb98088a9f7f678f57" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94fca0f2ef064318a6123861fb408f75" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f21bb0342114e388347a675ec9c6cfa" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 39449 } health_report { overall_health: HEALTHY } } }
W20260810 01:51:17.502585 7269 tablet.cc:2369] T ca3a5948333f47638b1353db4705a32f P 031521c20ae64fcb98088a9f7f678f57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 01:51:17.509351 7481 tablet.cc:2369] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 01:51:17.514940 7500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:17.510805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52102 (local address 127.9.77.129:39449)
0810 01:51:17.511011 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:17.511016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:17.511032 (+ 16us) server_negotiation.cc:407] Connection header received
0810 01:51:17.511141 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:17.511143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:17.511210 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:17.511333 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:51:17.511649 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.512302 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.513244 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.513488 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.513611 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:17.513968 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:51:17.514502 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:51:17.514653 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:17.514699 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 01:51:17.514759 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 01:51:17.522536 7404 raft_consensus.cc:1261] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75 [term 1 FOLLOWER]: Refusing update from remote peer 9f21bb0342114e388347a675ec9c6cfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:51:17.522534 7199 raft_consensus.cc:1261] T ca3a5948333f47638b1353db4705a32f P 031521c20ae64fcb98088a9f7f678f57 [term 1 FOLLOWER]: Refusing update from remote peer 9f21bb0342114e388347a675ec9c6cfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:51:17.523257 7521 consensus_queue.cc:1035] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "94fca0f2ef064318a6123861fb408f75" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 35505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:51:17.523483 7511 consensus_queue.cc:1035] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "031521c20ae64fcb98088a9f7f678f57" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 37121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:51:17.542929 6801 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:51:17.543193 6801 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:51:17.546289 6801 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb441c6a61814b1693600e15ba18f293: Sending DeleteTablet for 3 replicas of tablet ca3a5948333f47638b1353db4705a32f
I20260810 01:51:17.547300 7376 tablet_service.cc:1514] Processing DeleteTablet for tablet ca3a5948333f47638b1353db4705a32f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:51:17 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:42798
I20260810 01:51:17.547286 7172 tablet_service.cc:1514] Processing DeleteTablet for tablet ca3a5948333f47638b1353db4705a32f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:51:17 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:50904
I20260810 01:51:17.548048 6969 tablet_service.cc:1514] Processing DeleteTablet for tablet ca3a5948333f47638b1353db4705a32f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:51:17 UTC) from {username='kudu', principal='kudu/127.9.77.190@KRBTEST.COM'} at 127.0.0.1:52096
I20260810 01:51:17.548298 7542 tablet_replica.cc:330] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75: stopping tablet replica
I20260810 01:51:17.548647 7543 tablet_replica.cc:330] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa: stopping tablet replica
I20260810 01:51:17.548767 7542 raft_consensus.cc:2229] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:51:17.549202 7543 raft_consensus.cc:2229] T ca3a5948333f47638b1353db4705a32f P 9f21bb0342114e388347a675ec9c6cfa [term 1 LEADER]: Raft consensus shutting down.
I20260810 01:51:17.549280 7542 raft_consensus.cc:2258] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:51:17.549762 7541 tablet_replica.cc:330] T ca3a5948333f47638b1353db4705a32f P 031521c20ae64fcb98088a9f7f678f57: stopping tablet replica
I20260810 01:51:17.550091 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 6883
I20260810 01:51:17.552263 7542 ts_tablet_manager.cc:1905] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:51:17.556267 7542 ts_tablet_manager.cc:1918] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:51:17.556382 7542 log.cc:1196] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-2/wal/wals/ca3a5948333f47638b1353db4705a32f
I20260810 01:51:17.556728 7542 ts_tablet_manager.cc:1939] T ca3a5948333f47638b1353db4705a32f P 94fca0f2ef064318a6123861fb408f75: Deleting consensus metadata
I20260810 01:51:17.558395 6787 catalog_manager.cc:4841] TS 94fca0f2ef064318a6123861fb408f75 (127.9.77.131:35505): tablet ca3a5948333f47638b1353db4705a32f (table test-table [id=768f95c393b942a0ac64156dc9f27fb7]) successfully deleted
I20260810 01:51:17.558713 7541 raft_consensus.cc:2229] T ca3a5948333f47638b1353db4705a32f P 031521c20ae64fcb98088a9f7f678f57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:51:17.559151 7541 raft_consensus.cc:2258] T ca3a5948333f47638b1353db4705a32f P 031521c20ae64fcb98088a9f7f678f57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:51:17.570088 7541 ts_tablet_manager.cc:1905] T ca3a5948333f47638b1353db4705a32f P 031521c20ae64fcb98088a9f7f678f57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:51:17.574885 7541 ts_tablet_manager.cc:1918] T ca3a5948333f47638b1353db4705a32f P 031521c20ae64fcb98088a9f7f678f57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:51:17.575045 7541 log.cc:1196] T ca3a5948333f47638b1353db4705a32f P 031521c20ae64fcb98088a9f7f678f57: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786326392877349-9526-0/minicluster-data/ts-1/wal/wals/ca3a5948333f47638b1353db4705a32f
I20260810 01:51:17.575454 7541 ts_tablet_manager.cc:1939] T ca3a5948333f47638b1353db4705a32f P 031521c20ae64fcb98088a9f7f678f57: Deleting consensus metadata
W20260810 01:51:17.577075 6786 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.77.129:39449 (error 108)
I20260810 01:51:17.577363 6785 catalog_manager.cc:4841] TS 031521c20ae64fcb98088a9f7f678f57 (127.9.77.130:37121): tablet ca3a5948333f47638b1353db4705a32f (table test-table [id=768f95c393b942a0ac64156dc9f27fb7]) successfully deleted
I20260810 01:51:17.578243 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 7076
I20260810 01:51:17.578517 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:17.578092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:39449 (local address 127.0.0.1:52118)
0810 01:51:17.578271 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:17.578374 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.129:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260810 01:51:17.587524 6786 catalog_manager.cc:4584] TS 9f21bb0342114e388347a675ec9c6cfa (127.9.77.129:39449): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ca3a5948333f47638b1353db4705a32f: Network error: Client connection negotiation failed: client connection to 127.9.77.129:39449: connect: Connection refused (error 111)
I20260810 01:51:17.594913 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 7274
I20260810 01:51:17.608887 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 6764
2026-08-10T01:51:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2888 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T01:51:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:51:17Z Disabled control of system clock
I20260810 01:51:17.654817 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:42533
--webserver_interface=127.9.77.190
--webserver_port=0
--builtin_ntp_servers=127.9.77.148:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:42533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 01:51:17.791333 7550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:17.791688 7550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:17.791728 7550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:17.796002 7550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:17.796103 7550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:17.796118 7550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:17.796139 7550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:17.796154 7550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:17.801622 7550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:42533
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:42533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:17.803253 7550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:17.805230 7550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:17.815061 7561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:17.815114 7559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:17.815097 7558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:17.815515 7550 server_base.cc:1034] running on GCE node
I20260810 01:51:17.816357 7550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:17.817675 7550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:17.818845 7550 hybrid_clock.cc:648] HybridClock initialized: now 1786326677818830 us; error 32 us; skew 500 ppm
I20260810 01:51:17.821202 7550 webserver.cc:460] Webserver started at http://127.9.77.190:45669/ using document root <none> and password file <none>
I20260810 01:51:17.821939 7550 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:17.822010 7550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:17.822289 7550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:17.824909 7550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "d57f77539af34a6ba22887c5e77d316e"
format_stamp: "Formatted at 2026-08-10 01:51:17 on dist-test-slave-kvfs"
server_key: "7bccf00f481f8429cf5a3cbe79702da2"
server_key_iv: "fb13b311430137d07fa974a9012db60a"
server_key_version: "encryptionkey@0"
I20260810 01:51:17.825606 7550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "d57f77539af34a6ba22887c5e77d316e"
format_stamp: "Formatted at 2026-08-10 01:51:17 on dist-test-slave-kvfs"
server_key: "7bccf00f481f8429cf5a3cbe79702da2"
server_key_iv: "fb13b311430137d07fa974a9012db60a"
server_key_version: "encryptionkey@0"
I20260810 01:51:17.830217 7550 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 01:51:17.833207 7572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:17.834781 7550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:51:17.834970 7550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "d57f77539af34a6ba22887c5e77d316e"
format_stamp: "Formatted at 2026-08-10 01:51:17 on dist-test-slave-kvfs"
server_key: "7bccf00f481f8429cf5a3cbe79702da2"
server_key_iv: "fb13b311430137d07fa974a9012db60a"
server_key_version: "encryptionkey@0"
I20260810 01:51:17.835110 7550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:17.862907 7550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:17.863906 7550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:17.864185 7550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:17.873277 7550 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:42533
I20260810 01:51:17.873296 7653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:42533 every 8 connection(s)
I20260810 01:51:17.874529 7550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:51:17.878224 7654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:17.883057 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 7550
I20260810 01:51:17.883247 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:51:17.883577 9526 external_mini_cluster.cc:1428] Setting key 51e6da256235ae03e5701694535a0788
I20260810 01:51:17.885789 7654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d57f77539af34a6ba22887c5e77d316e: Bootstrap starting.
I20260810 01:51:17.888767 7654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d57f77539af34a6ba22887c5e77d316e: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:17.889710 7654 log.cc:824] T 00000000000000000000000000000000 P d57f77539af34a6ba22887c5e77d316e: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:17.892278 7654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d57f77539af34a6ba22887c5e77d316e: No bootstrap required, opened a new log
I20260810 01:51:17.895450 7657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:17.885495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47490 (local address 127.9.77.190:42533)
0810 01:51:17.886342 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:17.886362 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:17.886408 (+ 46us) server_negotiation.cc:407] Connection header received
0810 01:51:17.887199 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:17.887231 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:17.887781 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:17.888244 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:17.888793 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.890014 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.891010 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:17.891560 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:17.891764 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:17.892873 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:17.892879 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:17.892899 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:17.893407 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:17.893481 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:17.893504 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:17.893624 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:17.893825 (+ 201us) server_negotiation.cc:299] Negotiation successful
0810 01:51:17.894159 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":158,"threads_started":1}
I20260810 01:51:17.896216 7654 raft_consensus.cc:348] T 00000000000000000000000000000000 P d57f77539af34a6ba22887c5e77d316e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57f77539af34a6ba22887c5e77d316e" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 42533 } }
I20260810 01:51:17.896524 7654 raft_consensus.cc:374] T 00000000000000000000000000000000 P d57f77539af34a6ba22887c5e77d316e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:17.896608 7654 raft_consensus.cc:729] T 00000000000000000000000000000000 P d57f77539af34a6ba22887c5e77d316e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d57f77539af34a6ba22887c5e77d316e, State: Initialized, Role: FOLLOWER
I20260810 01:51:17.897142 7654 consensus_queue.cc:260] T 00000000000000000000000000000000 P d57f77539af34a6ba22887c5e77d316e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57f77539af34a6ba22887c5e77d316e" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 42533 } }
I20260810 01:51:17.897311 7654 raft_consensus.cc:388] T 00000000000000000000000000000000 P d57f77539af34a6ba22887c5e77d316e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:51:17.897401 7654 raft_consensus.cc:482] T 00000000000000000000000000000000 P d57f77539af34a6ba22887c5e77d316e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:51:17.897537 7654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d57f77539af34a6ba22887c5e77d316e [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:17.898820 7654 raft_consensus.cc:504] T 00000000000000000000000000000000 P d57f77539af34a6ba22887c5e77d316e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57f77539af34a6ba22887c5e77d316e" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 42533 } }
I20260810 01:51:17.899228 7654 leader_election.cc:302] T 00000000000000000000000000000000 P d57f77539af34a6ba22887c5e77d316e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d57f77539af34a6ba22887c5e77d316e; no voters:
I20260810 01:51:17.899593 7654 leader_election.cc:288] T 00000000000000000000000000000000 P d57f77539af34a6ba22887c5e77d316e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:51:17.899729 7659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d57f77539af34a6ba22887c5e77d316e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:17.900002 7659 raft_consensus.cc:686] T 00000000000000000000000000000000 P d57f77539af34a6ba22887c5e77d316e [term 1 LEADER]: Becoming Leader. State: Replica: d57f77539af34a6ba22887c5e77d316e, State: Running, Role: LEADER
I20260810 01:51:17.900384 7659 consensus_queue.cc:237] T 00000000000000000000000000000000 P d57f77539af34a6ba22887c5e77d316e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57f77539af34a6ba22887c5e77d316e" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 42533 } }
I20260810 01:51:17.901098 7654 sys_catalog.cc:564] T 00000000000000000000000000000000 P d57f77539af34a6ba22887c5e77d316e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:17.902544 7662 sys_catalog.cc:455] T 00000000000000000000000000000000 P d57f77539af34a6ba22887c5e77d316e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d57f77539af34a6ba22887c5e77d316e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57f77539af34a6ba22887c5e77d316e" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 42533 } } }
I20260810 01:51:17.902706 7662 sys_catalog.cc:458] T 00000000000000000000000000000000 P d57f77539af34a6ba22887c5e77d316e [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:17.903004 7666 sys_catalog.cc:455] T 00000000000000000000000000000000 P d57f77539af34a6ba22887c5e77d316e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d57f77539af34a6ba22887c5e77d316e. Latest consensus state: current_term: 1 leader_uuid: "d57f77539af34a6ba22887c5e77d316e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57f77539af34a6ba22887c5e77d316e" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 42533 } } }
I20260810 01:51:17.903106 7666 sys_catalog.cc:458] T 00000000000000000000000000000000 P d57f77539af34a6ba22887c5e77d316e [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:17.903295 7678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:51:17.907662 7678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:51:17.915239 7678 catalog_manager.cc:1347] Generated new cluster ID: a142d13e2e5342fe8a4e4125f7dfad80
I20260810 01:51:17.915349 7678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:51:17.921096 7678 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:51:17.922609 7678 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:51:17.931093 7678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d57f77539af34a6ba22887c5e77d316e: Generated new TSK 0
I20260810 01:51:17.932330 7678 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:51:17.942206 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.129:0
--local_ip_for_outbound_sockets=127.9.77.129
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:42533
--builtin_ntp_servers=127.9.77.148:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 01:51:18.082261 7691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:18.082607 7691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:18.082648 7691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:18.086862 7691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:18.086941 7691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:18.087038 7691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.129
I20260810 01:51:18.092057 7691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.77.129
--webserver_port=0
--tserver_master_addrs=127.9.77.190:42533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.129
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:18.093596 7691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:18.095465 7691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:18.108126 7697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:18.108111 7698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:18.108106 7701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:18.108513 7691 server_base.cc:1034] running on GCE node
I20260810 01:51:18.109063 7691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:18.109946 7691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:18.111156 7691 hybrid_clock.cc:648] HybridClock initialized: now 1786326678111117 us; error 52 us; skew 500 ppm
I20260810 01:51:18.113787 7691 webserver.cc:460] Webserver started at http://127.9.77.129:44919/ using document root <none> and password file <none>
I20260810 01:51:18.114593 7691 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:18.114704 7691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:18.115087 7691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:18.118185 7691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-0/data/instance:
uuid: "540437718a2347d2846c129af94db8f9"
format_stamp: "Formatted at 2026-08-10 01:51:18 on dist-test-slave-kvfs"
server_key: "5c21720a6457dbc3755e467c57e9ed38"
server_key_iv: "e8c63be4ed4030d7f450acb2a2240f7f"
server_key_version: "encryptionkey@0"
I20260810 01:51:18.118937 7691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance:
uuid: "540437718a2347d2846c129af94db8f9"
format_stamp: "Formatted at 2026-08-10 01:51:18 on dist-test-slave-kvfs"
server_key: "5c21720a6457dbc3755e467c57e9ed38"
server_key_iv: "e8c63be4ed4030d7f450acb2a2240f7f"
server_key_version: "encryptionkey@0"
I20260810 01:51:18.123842 7691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 01:51:18.127239 7708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:18.128979 7691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:51:18.129149 7691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-0/wal
uuid: "540437718a2347d2846c129af94db8f9"
format_stamp: "Formatted at 2026-08-10 01:51:18 on dist-test-slave-kvfs"
server_key: "5c21720a6457dbc3755e467c57e9ed38"
server_key_iv: "e8c63be4ed4030d7f450acb2a2240f7f"
server_key_version: "encryptionkey@0"
I20260810 01:51:18.129300 7691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:18.145622 7691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:18.146603 7691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:18.146860 7691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:18.147680 7691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:51:18.149036 7691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:51:18.149127 7691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:18.149230 7691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:51:18.149281 7691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:18.160120 7691 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.129:37309
I20260810 01:51:18.160224 7864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.129:37309 every 8 connection(s)
I20260810 01:51:18.161412 7691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-0/data/info.pb
I20260810 01:51:18.163264 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 7691
I20260810 01:51:18.163419 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-0/wal/instance
I20260810 01:51:18.163841 9526 external_mini_cluster.cc:1428] Setting key 760b58204e7df1e95f746c567dc3c712
I20260810 01:51:18.166440 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.130:0
--local_ip_for_outbound_sockets=127.9.77.130
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:42533
--builtin_ntp_servers=127.9.77.148:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 01:51:18.174002 7657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:18.163826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.129:56929 (local address 127.9.77.190:42533)
0810 01:51:18.164011 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:18.164040 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:18.165507 (+ 1467us) server_negotiation.cc:407] Connection header received
0810 01:51:18.167310 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:18.167313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:18.167382 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:18.167492 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:18.168771 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.169459 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.171349 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.171612 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.171711 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:18.173358 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:18.173361 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:18.173364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:18.173382 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:18.173410 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:18.173421 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:18.173505 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:18.173755 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 01:51:18.173834 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 01:51:18.175616 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:18.164516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:42533 (local address 127.9.77.129:56929)
0810 01:51:18.165325 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:18.165375 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:18.167052 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:18.167710 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:18.167719 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:18.167832 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:18.168560 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:18.168577 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.169621 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.169628 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:18.171194 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:18.171207 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.171804 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.171811 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:18.172118 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:18.172959 (+ 841us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 01:51:18.172964 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:18.173033 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 01:51:18.173145 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:18.173537 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:18.173553 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:18.173564 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 01:51:18.173737 (+ 173us) client_negotiation.cc:240] Negotiation successful
0810 01:51:18.174146 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":127,"threads_started":1}
I20260810 01:51:18.177336 7865 heartbeater.cc:347] Connected to a master server at 127.9.77.190:42533
I20260810 01:51:18.177675 7865 heartbeater.cc:464] Registering TS with master...
I20260810 01:51:18.178404 7865 heartbeater.cc:511] Master 127.9.77.190:42533 requested a full tablet report, sending...
I20260810 01:51:18.180446 7603 ts_manager.cc:194] Registered new tserver with Master: 540437718a2347d2846c129af94db8f9 (127.9.77.129:37309)
I20260810 01:51:18.181926 7603 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.77.129:56929
W20260810 01:51:18.306509 7871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:18.306875 7871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:18.306919 7871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:18.311553 7871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:18.311654 7871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:18.311762 7871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.130
I20260810 01:51:18.317875 7871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.77.130
--webserver_port=0
--tserver_master_addrs=127.9.77.190:42533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.130
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:18.319451 7871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:18.321425 7871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:18.332988 7877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:18.333057 7871 server_base.cc:1034] running on GCE node
W20260810 01:51:18.332999 7876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:18.332986 7883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:18.333699 7871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:18.334625 7871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:18.335793 7871 hybrid_clock.cc:648] HybridClock initialized: now 1786326678335741 us; error 67 us; skew 500 ppm
I20260810 01:51:18.338300 7871 webserver.cc:460] Webserver started at http://127.9.77.130:38339/ using document root <none> and password file <none>
I20260810 01:51:18.339066 7871 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:18.339191 7871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:18.339452 7871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:18.342180 7871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-1/data/instance:
uuid: "a5ca8fc2c2204dd8ba27742f4dc190ea"
format_stamp: "Formatted at 2026-08-10 01:51:18 on dist-test-slave-kvfs"
server_key: "31b58d151864df14d07e4a08ed50e910"
server_key_iv: "722ad636fe3453e501d473ef125c9400"
server_key_version: "encryptionkey@0"
I20260810 01:51:18.342826 7871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance:
uuid: "a5ca8fc2c2204dd8ba27742f4dc190ea"
format_stamp: "Formatted at 2026-08-10 01:51:18 on dist-test-slave-kvfs"
server_key: "31b58d151864df14d07e4a08ed50e910"
server_key_iv: "722ad636fe3453e501d473ef125c9400"
server_key_version: "encryptionkey@0"
I20260810 01:51:18.347426 7871 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 01:51:18.350415 7891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:18.352133 7871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:51:18.352273 7871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-1/wal
uuid: "a5ca8fc2c2204dd8ba27742f4dc190ea"
format_stamp: "Formatted at 2026-08-10 01:51:18 on dist-test-slave-kvfs"
server_key: "31b58d151864df14d07e4a08ed50e910"
server_key_iv: "722ad636fe3453e501d473ef125c9400"
server_key_version: "encryptionkey@0"
I20260810 01:51:18.352408 7871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:18.368984 7871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:18.369910 7871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:18.370157 7871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:18.370947 7871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:51:18.372241 7871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:51:18.372315 7871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:18.372421 7871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:51:18.372463 7871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:18.383128 7871 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.130:41423
I20260810 01:51:18.383165 8048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.130:41423 every 8 connection(s)
I20260810 01:51:18.384406 7871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-1/data/info.pb
I20260810 01:51:18.388413 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 7871
I20260810 01:51:18.388594 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-1/wal/instance
I20260810 01:51:18.388980 9526 external_mini_cluster.cc:1428] Setting key 1b9fa73f324ef53efa546022c77ac33a
I20260810 01:51:18.391534 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.77.131:0
--local_ip_for_outbound_sockets=127.9.77.131
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:42533
--builtin_ntp_servers=127.9.77.148:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 01:51:18.396613 7657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:18.386878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.130:58789 (local address 127.9.77.190:42533)
0810 01:51:18.387059 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:18.387063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:18.388462 (+ 1399us) server_negotiation.cc:407] Connection header received
0810 01:51:18.390347 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:18.390350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:18.390403 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:18.390481 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:18.391702 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.392397 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.393692 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.394099 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.394228 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:18.395903 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:18.395910 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:18.395916 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:18.395937 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:18.395970 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:18.395984 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:18.396102 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:18.396271 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 01:51:18.396359 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 01:51:18.397650 8052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:18.387379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:42533 (local address 127.9.77.130:58789)
0810 01:51:18.388274 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:18.388325 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:18.390085 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:18.390744 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:18.390755 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:18.390894 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:18.391540 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:18.391556 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.392550 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.392553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:18.393558 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:18.393568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.394314 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.394320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:18.394639 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:18.395582 (+ 943us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 01:51:18.395583 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:18.395638 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 01:51:18.395723 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:18.396109 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:18.396117 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:18.396122 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 01:51:18.396242 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 01:51:18.396547 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":135,"threads_started":1}
I20260810 01:51:18.399152 8049 heartbeater.cc:347] Connected to a master server at 127.9.77.190:42533
I20260810 01:51:18.399463 8049 heartbeater.cc:464] Registering TS with master...
I20260810 01:51:18.400233 8049 heartbeater.cc:511] Master 127.9.77.190:42533 requested a full tablet report, sending...
I20260810 01:51:18.401680 7603 ts_manager.cc:194] Registered new tserver with Master: a5ca8fc2c2204dd8ba27742f4dc190ea (127.9.77.130:41423)
I20260810 01:51:18.402432 7603 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.77.130:58789
W20260810 01:51:18.533985 8053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:18.534315 8053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:18.534356 8053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:18.538669 8053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:18.538748 8053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:18.538810 8053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.77.131
I20260810 01:51:18.544123 8053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.77.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.77.131
--webserver_port=0
--tserver_master_addrs=127.9.77.190:42533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.77.131
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:18.545636 8053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:18.547559 8053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:18.559728 8062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:18.559728 8060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:18.559742 8064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:18.560077 8053 server_base.cc:1034] running on GCE node
I20260810 01:51:18.560623 8053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:18.561563 8053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:18.562767 8053 hybrid_clock.cc:648] HybridClock initialized: now 1786326678562745 us; error 47 us; skew 500 ppm
I20260810 01:51:18.565062 8053 webserver.cc:460] Webserver started at http://127.9.77.131:37831/ using document root <none> and password file <none>
I20260810 01:51:18.565810 8053 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:18.565912 8053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:18.566201 8053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:18.568956 8053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-2/data/instance:
uuid: "a85b601d720a4699891f9cb436da0521"
format_stamp: "Formatted at 2026-08-10 01:51:18 on dist-test-slave-kvfs"
server_key: "71b8188daee28a9b1577796bb31da29a"
server_key_iv: "81b385d5f083301f928f92e39148160c"
server_key_version: "encryptionkey@0"
I20260810 01:51:18.569728 8053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance:
uuid: "a85b601d720a4699891f9cb436da0521"
format_stamp: "Formatted at 2026-08-10 01:51:18 on dist-test-slave-kvfs"
server_key: "71b8188daee28a9b1577796bb31da29a"
server_key_iv: "81b385d5f083301f928f92e39148160c"
server_key_version: "encryptionkey@0"
I20260810 01:51:18.574685 8053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 01:51:18.577847 8073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:18.579523 8053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 01:51:18.579694 8053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-2/wal
uuid: "a85b601d720a4699891f9cb436da0521"
format_stamp: "Formatted at 2026-08-10 01:51:18 on dist-test-slave-kvfs"
server_key: "71b8188daee28a9b1577796bb31da29a"
server_key_iv: "81b385d5f083301f928f92e39148160c"
server_key_version: "encryptionkey@0"
I20260810 01:51:18.579851 8053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:18.602944 8053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:18.604125 8053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:18.604403 8053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:18.605258 8053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 01:51:18.606627 8053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 01:51:18.606709 8053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:18.606808 8053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 01:51:18.606858 8053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:18.618461 8053 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.131:37369
I20260810 01:51:18.618494 8254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.131:37369 every 8 connection(s)
I20260810 01:51:18.619711 8053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-2/data/info.pb
I20260810 01:51:18.622817 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 8053
I20260810 01:51:18.622987 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-2/wal/instance
I20260810 01:51:18.623435 9526 external_mini_cluster.cc:1428] Setting key 5b9232a784c8a0b13f5d5341993788b0
I20260810 01:51:18.631824 7657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:18.622305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:34517 (local address 127.9.77.190:42533)
0810 01:51:18.622503 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:18.622508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:18.624107 (+ 1599us) server_negotiation.cc:407] Connection header received
0810 01:51:18.626337 (+ 2230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:18.626339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:18.626397 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:18.626488 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:18.627705 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.628403 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.629684 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.629928 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.630018 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:18.631349 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:18.631351 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:18.631354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:18.631366 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:18.631386 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:18.631394 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:18.631455 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:18.631609 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 01:51:18.631672 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 01:51:18.632985 8257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:18.622932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:42533 (local address 127.9.77.131:34517)
0810 01:51:18.623908 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:18.623975 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:18.626067 (+ 2092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:18.626749 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:18.626763 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:18.626878 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:18.627560 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:18.627575 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.628536 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.628540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:18.629567 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:18.629577 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.630093 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.630096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:18.630260 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:18.631075 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 01:51:18.631077 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:18.631128 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 01:51:18.631209 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:18.631471 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:18.631479 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:18.631484 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 01:51:18.631596 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 01:51:18.631884 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":201,"threads_started":1}
I20260810 01:51:18.634461 8255 heartbeater.cc:347] Connected to a master server at 127.9.77.190:42533
I20260810 01:51:18.634760 8255 heartbeater.cc:464] Registering TS with master...
I20260810 01:51:18.635538 8255 heartbeater.cc:511] Master 127.9.77.190:42533 requested a full tablet report, sending...
I20260810 01:51:18.636969 7603 ts_manager.cc:194] Registered new tserver with Master: a85b601d720a4699891f9cb436da0521 (127.9.77.131:37369)
I20260810 01:51:18.637786 7603 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.77.131:34517
I20260810 01:51:18.638131 9526 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 01:51:18.645931 7657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:18.640953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47498 (local address 127.9.77.190:42533)
0810 01:51:18.641168 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:18.641173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:18.641438 (+ 265us) server_negotiation.cc:407] Connection header received
0810 01:51:18.641626 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:18.641629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:18.641691 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:18.641824 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:18.642166 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.642792 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.643807 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.644156 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.644291 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:18.645364 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:18.645368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:18.645371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:18.645388 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:18.645420 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:18.645430 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:18.645522 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:18.645660 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 01:51:18.645741 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 01:51:18.649830 7603 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 01:51:18.653870 7603 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 01:51:18.672256 8276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:18.665160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.131:37369 (local address 127.0.0.1:39068)
0810 01:51:18.665776 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:18.665819 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:18.665987 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:18.667253 (+ 1266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:18.667256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:18.667270 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:18.667368 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:18.667374 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.668513 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.668520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:18.670177 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:18.670189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.671471 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.671475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:18.671685 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:18.671737 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 01:51:18.671875 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 01:51:18.672051 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":170,"spinlock_wait_cycles":16000,"thread_start_us":138,"threads_started":1}
I20260810 01:51:18.672416 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:18.665416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39068 (local address 127.9.77.131:37369)
0810 01:51:18.666506 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:18.666519 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:18.666547 (+ 28us) server_negotiation.cc:407] Connection header received
0810 01:51:18.666637 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:18.666642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:18.666938 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:18.667105 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:18.667510 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.668391 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.670334 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.671280 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.671407 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:18.671658 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:18.671957 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 01:51:18.672153 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":969,"thread_start_us":150,"threads_started":1}
I20260810 01:51:18.673321 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:18.666174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46592 (local address 127.9.77.129:37309)
0810 01:51:18.666637 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:18.666644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:18.666667 (+ 23us) server_negotiation.cc:407] Connection header received
0810 01:51:18.666753 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:18.666758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:18.666941 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:18.667091 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:18.667376 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.668668 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.671732 (+ 3064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.672633 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.672761 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:18.672981 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:18.673042 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 01:51:18.673159 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":124,"threads_started":1}
I20260810 01:51:18.673317 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:18.665631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:37309 (local address 127.0.0.1:46592)
0810 01:51:18.665892 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:18.665905 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:18.665990 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:18.667128 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:18.667131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:18.667148 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:18.667258 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:18.667265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.668791 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.668795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:18.670187 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:18.670193 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.672750 (+ 2557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.672754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:18.672900 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:18.672931 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 01:51:18.673008 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 01:51:18.673134 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":81,"thread_start_us":111,"threads_started":1}
I20260810 01:51:18.673812 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:18.664779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:41423 (local address 127.0.0.1:33350)
0810 01:51:18.665744 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:18.665807 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:18.666277 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:18.666921 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:18.666928 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:18.666953 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:18.667170 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:18.667179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.669015 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.669021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:18.670202 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:18.670212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.671471 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.671475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:18.673493 (+ 2018us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:18.673519 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 01:51:18.673601 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 01:51:18.673675 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":1863,"spinlock_wait_cycles":6912,"thread_start_us":117,"threads_started":1}
I20260810 01:51:18.674083 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:18.665003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33350 (local address 127.9.77.130:41423)
0810 01:51:18.665617 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:18.665632 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:18.666246 (+ 614us) server_negotiation.cc:407] Connection header received
0810 01:51:18.666478 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:18.666486 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:18.666738 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:18.666910 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:18.667295 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.668668 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.670393 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.671280 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.671407 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:18.671658 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:18.673680 (+ 2022us) server_negotiation.cc:299] Negotiation successful
0810 01:51:18.673811 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":176,"threads_started":1}
I20260810 01:51:18.677007 8151 tablet_service.cc:1467] Processing CreateTablet for tablet 660d9861433145048e92a1298cebdf45 (DEFAULT_TABLE table=test-table [id=d1de78e7359f497eb4bdbf9b321b9b04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:51:18.676992 7772 tablet_service.cc:1467] Processing CreateTablet for tablet 660d9861433145048e92a1298cebdf45 (DEFAULT_TABLE table=test-table [id=d1de78e7359f497eb4bdbf9b321b9b04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:51:18.677539 7960 tablet_service.cc:1467] Processing CreateTablet for tablet 660d9861433145048e92a1298cebdf45 (DEFAULT_TABLE table=test-table [id=d1de78e7359f497eb4bdbf9b321b9b04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 01:51:18.678424 8151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660d9861433145048e92a1298cebdf45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:18.678427 7772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660d9861433145048e92a1298cebdf45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:18.678486 7960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 660d9861433145048e92a1298cebdf45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:18.686592 8283 tablet_bootstrap.cc:492] T 660d9861433145048e92a1298cebdf45 P 540437718a2347d2846c129af94db8f9: Bootstrap starting.
I20260810 01:51:18.687310 8284 tablet_bootstrap.cc:492] T 660d9861433145048e92a1298cebdf45 P a5ca8fc2c2204dd8ba27742f4dc190ea: Bootstrap starting.
I20260810 01:51:18.688422 8285 tablet_bootstrap.cc:492] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521: Bootstrap starting.
I20260810 01:51:18.690187 8283 tablet_bootstrap.cc:654] T 660d9861433145048e92a1298cebdf45 P 540437718a2347d2846c129af94db8f9: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:18.690189 8284 tablet_bootstrap.cc:654] T 660d9861433145048e92a1298cebdf45 P a5ca8fc2c2204dd8ba27742f4dc190ea: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:18.690539 8285 tablet_bootstrap.cc:654] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:18.691211 8283 log.cc:824] T 660d9861433145048e92a1298cebdf45 P 540437718a2347d2846c129af94db8f9: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:18.691213 8284 log.cc:824] T 660d9861433145048e92a1298cebdf45 P a5ca8fc2c2204dd8ba27742f4dc190ea: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:18.691282 8285 log.cc:824] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:18.693516 8284 tablet_bootstrap.cc:492] T 660d9861433145048e92a1298cebdf45 P a5ca8fc2c2204dd8ba27742f4dc190ea: No bootstrap required, opened a new log
I20260810 01:51:18.693512 8285 tablet_bootstrap.cc:492] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521: No bootstrap required, opened a new log
I20260810 01:51:18.693754 8284 ts_tablet_manager.cc:1397] T 660d9861433145048e92a1298cebdf45 P a5ca8fc2c2204dd8ba27742f4dc190ea: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 01:51:18.693758 8285 ts_tablet_manager.cc:1397] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 01:51:18.697687 8283 tablet_bootstrap.cc:492] T 660d9861433145048e92a1298cebdf45 P 540437718a2347d2846c129af94db8f9: No bootstrap required, opened a new log
I20260810 01:51:18.697930 8283 ts_tablet_manager.cc:1397] T 660d9861433145048e92a1298cebdf45 P 540437718a2347d2846c129af94db8f9: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260810 01:51:18.698055 8285 raft_consensus.cc:348] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca8fc2c2204dd8ba27742f4dc190ea" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 41423 } } peers { permanent_uuid: "540437718a2347d2846c129af94db8f9" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37309 } } peers { permanent_uuid: "a85b601d720a4699891f9cb436da0521" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 37369 } }
I20260810 01:51:18.698378 8285 raft_consensus.cc:374] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:18.698208 8284 raft_consensus.cc:348] T 660d9861433145048e92a1298cebdf45 P a5ca8fc2c2204dd8ba27742f4dc190ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca8fc2c2204dd8ba27742f4dc190ea" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 41423 } } peers { permanent_uuid: "540437718a2347d2846c129af94db8f9" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37309 } } peers { permanent_uuid: "a85b601d720a4699891f9cb436da0521" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 37369 } }
I20260810 01:51:18.698438 8285 raft_consensus.cc:729] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a85b601d720a4699891f9cb436da0521, State: Initialized, Role: FOLLOWER
I20260810 01:51:18.698462 8284 raft_consensus.cc:374] T 660d9861433145048e92a1298cebdf45 P a5ca8fc2c2204dd8ba27742f4dc190ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:18.698513 8284 raft_consensus.cc:729] T 660d9861433145048e92a1298cebdf45 P a5ca8fc2c2204dd8ba27742f4dc190ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ca8fc2c2204dd8ba27742f4dc190ea, State: Initialized, Role: FOLLOWER
I20260810 01:51:18.699008 8285 consensus_queue.cc:260] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca8fc2c2204dd8ba27742f4dc190ea" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 41423 } } peers { permanent_uuid: "540437718a2347d2846c129af94db8f9" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37309 } } peers { permanent_uuid: "a85b601d720a4699891f9cb436da0521" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 37369 } }
I20260810 01:51:18.699028 8284 consensus_queue.cc:260] T 660d9861433145048e92a1298cebdf45 P a5ca8fc2c2204dd8ba27742f4dc190ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca8fc2c2204dd8ba27742f4dc190ea" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 41423 } } peers { permanent_uuid: "540437718a2347d2846c129af94db8f9" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37309 } } peers { permanent_uuid: "a85b601d720a4699891f9cb436da0521" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 37369 } }
I20260810 01:51:18.699846 8255 heartbeater.cc:503] Master 127.9.77.190:42533 was elected leader, sending a full tablet report...
I20260810 01:51:18.700312 8285 ts_tablet_manager.cc:1428] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260810 01:51:18.700315 8284 ts_tablet_manager.cc:1428] T 660d9861433145048e92a1298cebdf45 P a5ca8fc2c2204dd8ba27742f4dc190ea: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260810 01:51:18.700701 8049 heartbeater.cc:503] Master 127.9.77.190:42533 was elected leader, sending a full tablet report...
I20260810 01:51:18.702967 8283 raft_consensus.cc:348] T 660d9861433145048e92a1298cebdf45 P 540437718a2347d2846c129af94db8f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca8fc2c2204dd8ba27742f4dc190ea" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 41423 } } peers { permanent_uuid: "540437718a2347d2846c129af94db8f9" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37309 } } peers { permanent_uuid: "a85b601d720a4699891f9cb436da0521" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 37369 } }
I20260810 01:51:18.703202 8283 raft_consensus.cc:374] T 660d9861433145048e92a1298cebdf45 P 540437718a2347d2846c129af94db8f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:18.703293 8283 raft_consensus.cc:729] T 660d9861433145048e92a1298cebdf45 P 540437718a2347d2846c129af94db8f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 540437718a2347d2846c129af94db8f9, State: Initialized, Role: FOLLOWER
I20260810 01:51:18.703840 8283 consensus_queue.cc:260] T 660d9861433145048e92a1298cebdf45 P 540437718a2347d2846c129af94db8f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca8fc2c2204dd8ba27742f4dc190ea" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 41423 } } peers { permanent_uuid: "540437718a2347d2846c129af94db8f9" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37309 } } peers { permanent_uuid: "a85b601d720a4699891f9cb436da0521" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 37369 } }
I20260810 01:51:18.704591 7865 heartbeater.cc:503] Master 127.9.77.190:42533 was elected leader, sending a full tablet report...
I20260810 01:51:18.705323 8283 ts_tablet_manager.cc:1428] T 660d9861433145048e92a1298cebdf45 P 540437718a2347d2846c129af94db8f9: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 01:51:18.853645 8290 raft_consensus.cc:482] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 01:51:18.853988 8290 raft_consensus.cc:504] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca8fc2c2204dd8ba27742f4dc190ea" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 41423 } } peers { permanent_uuid: "540437718a2347d2846c129af94db8f9" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37309 } } peers { permanent_uuid: "a85b601d720a4699891f9cb436da0521" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 37369 } }
I20260810 01:51:18.855587 8290 leader_election.cc:288] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5ca8fc2c2204dd8ba27742f4dc190ea (127.9.77.130:41423), 540437718a2347d2846c129af94db8f9 (127.9.77.129:37309)
I20260810 01:51:18.860612 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:18.856144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:58151 (local address 127.9.77.130:41423)
0810 01:51:18.856321 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:18.856326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:18.856349 (+ 23us) server_negotiation.cc:407] Connection header received
0810 01:51:18.856444 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:18.856450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:18.856526 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:18.856628 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:18.857174 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.857988 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.859327 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.859957 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.860120 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:18.860172 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:18.860413 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 01:51:18.860481 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260810 01:51:18.860651 8257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:18.855969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.130:41423 (local address 127.9.77.131:58151)
0810 01:51:18.856158 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:18.856177 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:18.856306 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:18.856806 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:18.856809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:18.856832 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:18.856996 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:18.857004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.858112 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.858116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:18.859184 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:18.859193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.860120 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.860125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:18.860260 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:18.860293 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 01:51:18.860390 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 01:51:18.860483 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 01:51:18.861372 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:18.856330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:37309 (local address 127.9.77.131:53695)
0810 01:51:18.856945 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:18.856960 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:18.857074 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:18.857497 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:18.857501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:18.857520 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 01:51:18.857657 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:18.857662 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.858704 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.858706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:18.859673 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:18.859684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.860941 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.860944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:18.861057 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:18.861085 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 01:51:18.861169 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 01:51:18.861253 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":179,"threads_started":1}
I20260810 01:51:18.861483 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:18.856698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.77.131:53695 (local address 127.9.77.129:37309)
0810 01:51:18.856879 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:18.856885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:18.857071 (+ 186us) server_negotiation.cc:407] Connection header received
0810 01:51:18.857190 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:18.857193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:18.857259 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:18.857336 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 01:51:18.857830 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.858577 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.859862 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.860606 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.860800 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:18.860864 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:18.861237 (+ 373us) server_negotiation.cc:299] Negotiation successful
0810 01:51:18.861342 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260810 01:51:18.861641 7988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660d9861433145048e92a1298cebdf45" candidate_uuid: "a85b601d720a4699891f9cb436da0521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca8fc2c2204dd8ba27742f4dc190ea" is_pre_election: true
I20260810 01:51:18.862089 7988 raft_consensus.cc:2454] T 660d9861433145048e92a1298cebdf45 P a5ca8fc2c2204dd8ba27742f4dc190ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a85b601d720a4699891f9cb436da0521 in term 0.
I20260810 01:51:18.862200 7801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660d9861433145048e92a1298cebdf45" candidate_uuid: "a85b601d720a4699891f9cb436da0521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540437718a2347d2846c129af94db8f9" is_pre_election: true
I20260810 01:51:18.862586 7801 raft_consensus.cc:2454] T 660d9861433145048e92a1298cebdf45 P 540437718a2347d2846c129af94db8f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a85b601d720a4699891f9cb436da0521 in term 0.
I20260810 01:51:18.862658 8081 leader_election.cc:302] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5ca8fc2c2204dd8ba27742f4dc190ea, a85b601d720a4699891f9cb436da0521; no voters:
I20260810 01:51:18.863068 8290 raft_consensus.cc:2781] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 01:51:18.863188 8290 raft_consensus.cc:482] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 01:51:18.863257 8290 raft_consensus.cc:3037] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:18.864615 8290 raft_consensus.cc:504] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca8fc2c2204dd8ba27742f4dc190ea" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 41423 } } peers { permanent_uuid: "540437718a2347d2846c129af94db8f9" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37309 } } peers { permanent_uuid: "a85b601d720a4699891f9cb436da0521" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 37369 } }
I20260810 01:51:18.865010 8290 leader_election.cc:288] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521 [CANDIDATE]: Term 1 election: Requested vote from peers a5ca8fc2c2204dd8ba27742f4dc190ea (127.9.77.130:41423), 540437718a2347d2846c129af94db8f9 (127.9.77.129:37309)
I20260810 01:51:18.865420 7988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660d9861433145048e92a1298cebdf45" candidate_uuid: "a85b601d720a4699891f9cb436da0521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5ca8fc2c2204dd8ba27742f4dc190ea"
I20260810 01:51:18.865532 7801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "660d9861433145048e92a1298cebdf45" candidate_uuid: "a85b601d720a4699891f9cb436da0521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540437718a2347d2846c129af94db8f9"
I20260810 01:51:18.865576 7988 raft_consensus.cc:3037] T 660d9861433145048e92a1298cebdf45 P a5ca8fc2c2204dd8ba27742f4dc190ea [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:18.865710 7801 raft_consensus.cc:3037] T 660d9861433145048e92a1298cebdf45 P 540437718a2347d2846c129af94db8f9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:18.866858 7988 raft_consensus.cc:2454] T 660d9861433145048e92a1298cebdf45 P a5ca8fc2c2204dd8ba27742f4dc190ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a85b601d720a4699891f9cb436da0521 in term 1.
I20260810 01:51:18.866967 7801 raft_consensus.cc:2454] T 660d9861433145048e92a1298cebdf45 P 540437718a2347d2846c129af94db8f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a85b601d720a4699891f9cb436da0521 in term 1.
I20260810 01:51:18.867213 8081 leader_election.cc:302] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5ca8fc2c2204dd8ba27742f4dc190ea, a85b601d720a4699891f9cb436da0521; no voters:
I20260810 01:51:18.867429 8290 raft_consensus.cc:2781] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:18.867743 8290 raft_consensus.cc:686] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521 [term 1 LEADER]: Becoming Leader. State: Replica: a85b601d720a4699891f9cb436da0521, State: Running, Role: LEADER
I20260810 01:51:18.868254 8290 consensus_queue.cc:237] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca8fc2c2204dd8ba27742f4dc190ea" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 41423 } } peers { permanent_uuid: "540437718a2347d2846c129af94db8f9" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37309 } } peers { permanent_uuid: "a85b601d720a4699891f9cb436da0521" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 37369 } }
W20260810 01:51:18.870961 8256 tablet.cc:2369] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 01:51:18.872463 7600 catalog_manager.cc:5495] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521 reported cstate change: term changed from 0 to 1, leader changed from <none> to a85b601d720a4699891f9cb436da0521 (127.9.77.131). New cstate: current_term: 1 leader_uuid: "a85b601d720a4699891f9cb436da0521" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5ca8fc2c2204dd8ba27742f4dc190ea" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 41423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "540437718a2347d2846c129af94db8f9" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a85b601d720a4699891f9cb436da0521" member_type: VOTER last_known_addr { host: "127.9.77.131" port: 37369 } health_report { overall_health: HEALTHY } } }
W20260810 01:51:18.885895 8051 tablet.cc:2369] T 660d9861433145048e92a1298cebdf45 P a5ca8fc2c2204dd8ba27742f4dc190ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 01:51:18.913160 7867 tablet.cc:2369] T 660d9861433145048e92a1298cebdf45 P 540437718a2347d2846c129af94db8f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 01:51:18.913970 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:18.909657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.9.77.131:37369)
0810 01:51:18.909837 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:18.909842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:18.909855 (+ 13us) server_negotiation.cc:407] Connection header received
0810 01:51:18.909965 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:18.909967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:18.910040 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:18.910179 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:51:18.910464 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.911046 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.912001 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:18.912279 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:18.912399 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:18.912766 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:51:18.913532 (+ 766us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:51:18.913667 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:18.913733 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 01:51:18.913803 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 01:51:18.921530 7801 raft_consensus.cc:1261] T 660d9861433145048e92a1298cebdf45 P 540437718a2347d2846c129af94db8f9 [term 1 FOLLOWER]: Refusing update from remote peer a85b601d720a4699891f9cb436da0521: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:51:18.921530 7988 raft_consensus.cc:1261] T 660d9861433145048e92a1298cebdf45 P a5ca8fc2c2204dd8ba27742f4dc190ea [term 1 FOLLOWER]: Refusing update from remote peer a85b601d720a4699891f9cb436da0521: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:51:18.922461 8290 consensus_queue.cc:1035] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521 [LEADER]: Connected to new peer: Peer: permanent_uuid: "540437718a2347d2846c129af94db8f9" member_type: VOTER last_known_addr { host: "127.9.77.129" port: 37309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:51:18.922737 8293 consensus_queue.cc:1035] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5ca8fc2c2204dd8ba27742f4dc190ea" member_type: VOTER last_known_addr { host: "127.9.77.130" port: 41423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:51:18.944281 7603 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:51:18.944518 7603 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 01:51:18.947743 7603 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d57f77539af34a6ba22887c5e77d316e: Sending DeleteTablet for 3 replicas of tablet 660d9861433145048e92a1298cebdf45
I20260810 01:51:18.948953 8151 tablet_service.cc:1514] Processing DeleteTablet for tablet 660d9861433145048e92a1298cebdf45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:51:18 UTC) from {username='slave'} at 127.0.0.1:39068
I20260810 01:51:18.948985 7960 tablet_service.cc:1514] Processing DeleteTablet for tablet 660d9861433145048e92a1298cebdf45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:51:18 UTC) from {username='slave'} at 127.0.0.1:33350
I20260810 01:51:18.949688 8315 tablet_replica.cc:330] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521: stopping tablet replica
I20260810 01:51:18.950035 8313 tablet_replica.cc:330] T 660d9861433145048e92a1298cebdf45 P a5ca8fc2c2204dd8ba27742f4dc190ea: stopping tablet replica
I20260810 01:51:18.949954 7772 tablet_service.cc:1514] Processing DeleteTablet for tablet 660d9861433145048e92a1298cebdf45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 01:51:18 UTC) from {username='slave'} at 127.0.0.1:46592
I20260810 01:51:18.950307 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 7691
I20260810 01:51:18.950768 8315 raft_consensus.cc:2229] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521 [term 1 LEADER]: Raft consensus shutting down.
I20260810 01:51:18.951117 8313 raft_consensus.cc:2229] T 660d9861433145048e92a1298cebdf45 P a5ca8fc2c2204dd8ba27742f4dc190ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 01:51:18.951825 8315 raft_consensus.cc:2258] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:51:18.952320 8313 raft_consensus.cc:2258] T 660d9861433145048e92a1298cebdf45 P a5ca8fc2c2204dd8ba27742f4dc190ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 01:51:18.956959 8315 ts_tablet_manager.cc:1905] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:51:18.961831 8315 ts_tablet_manager.cc:1918] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:51:18.962085 8315 log.cc:1196] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-2/wal/wals/660d9861433145048e92a1298cebdf45
I20260810 01:51:18.962697 8315 ts_tablet_manager.cc:1939] T 660d9861433145048e92a1298cebdf45 P a85b601d720a4699891f9cb436da0521: Deleting consensus metadata
I20260810 01:51:18.971313 7577 catalog_manager.cc:4841] TS a85b601d720a4699891f9cb436da0521 (127.9.77.131:37369): tablet 660d9861433145048e92a1298cebdf45 (table test-table [id=d1de78e7359f497eb4bdbf9b321b9b04]) successfully deleted
I20260810 01:51:18.973328 8313 ts_tablet_manager.cc:1905] T 660d9861433145048e92a1298cebdf45 P a5ca8fc2c2204dd8ba27742f4dc190ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 01:51:18.976302 8313 ts_tablet_manager.cc:1918] T 660d9861433145048e92a1298cebdf45 P a5ca8fc2c2204dd8ba27742f4dc190ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 01:51:18.976461 8313 log.cc:1196] T 660d9861433145048e92a1298cebdf45 P a5ca8fc2c2204dd8ba27742f4dc190ea: Deleting WAL directory at /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786326392877349-9526-0/minicluster-data/ts-1/wal/wals/660d9861433145048e92a1298cebdf45
I20260810 01:51:18.976805 8313 ts_tablet_manager.cc:1939] T 660d9861433145048e92a1298cebdf45 P a5ca8fc2c2204dd8ba27742f4dc190ea: Deleting consensus metadata
I20260810 01:51:18.978574 7579 catalog_manager.cc:4841] TS a5ca8fc2c2204dd8ba27742f4dc190ea (127.9.77.130:41423): tablet 660d9861433145048e92a1298cebdf45 (table test-table [id=d1de78e7359f497eb4bdbf9b321b9b04]) successfully deleted
W20260810 01:51:18.980064 7579 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.77.129:37309 (error 108)
I20260810 01:51:18.980381 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 7871
I20260810 01:51:18.992381 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:18.990387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.129:37309 (local address 127.0.0.1:46606)
0810 01:51:18.992065 (+ 1678us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:18.992199 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.129:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1494}
W20260810 01:51:18.993484 7579 catalog_manager.cc:4584] TS 540437718a2347d2846c129af94db8f9 (127.9.77.129:37309): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 660d9861433145048e92a1298cebdf45: Network error: Client connection negotiation failed: client connection to 127.9.77.129:37309: connect: Connection refused (error 111)
I20260810 01:51:19.001835 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 8053
I20260810 01:51:19.020174 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 7550
2026-08-10T01:51:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1414 ms)
[----------] 6 tests from EncryptionPolicyTest (13069 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:51:19 dist-test-slave-kvfs krb5kdc[8324](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:51:19 dist-test-slave-kvfs krb5kdc[8324](info): set up 2 sockets
Aug 10 01:51:19 dist-test-slave-kvfs krb5kdc[8324](info): commencing operation
krb5kdc: starting...
W20260810 01:51:20.338449 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.283s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:51:20 dist-test-slave-kvfs krb5kdc[8324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:51:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:51:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 01:51:20.496585 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36731
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:20.637641 8350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:20.638108 8350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:20.638192 8350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:20.642582 8350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:20.642729 8350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:20.642767 8350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:20.642807 8350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:20.642850 8350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:20.649168 8350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36731
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:20.650832 8350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:20.652285 8350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:20.663290 8358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:20.663349 8362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:20.663408 8350 server_base.cc:1034] running on GCE node
W20260810 01:51:20.663290 8357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:20.664553 8350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:20.666064 8350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:20.667251 8350 hybrid_clock.cc:648] HybridClock initialized: now 1786326680667234 us; error 31 us; skew 500 ppm
Aug 10 01:51:20 dist-test-slave-kvfs krb5kdc[8324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:20.671155 8350 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 01:51:20.672484 8350 webserver.cc:460] Webserver started at http://127.0.0.1:41715/ using document root <none> and password file <none>
I20260810 01:51:20.673255 8350 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:20.673331 8350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:20.673575 8350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:20.676607 8350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "0994ffb71f304afb8f52cc2a30e3cdd3"
format_stamp: "Formatted at 2026-08-10 01:51:20 on dist-test-slave-kvfs"
server_key: "6163af4848b55a0d39348cd03bb8f33c"
server_key_iv: "4ba06952af8689cb86bec0c21fe94182"
server_key_version: "encryptionkey@0"
I20260810 01:51:20.677369 8350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "0994ffb71f304afb8f52cc2a30e3cdd3"
format_stamp: "Formatted at 2026-08-10 01:51:20 on dist-test-slave-kvfs"
server_key: "6163af4848b55a0d39348cd03bb8f33c"
server_key_iv: "4ba06952af8689cb86bec0c21fe94182"
server_key_version: "encryptionkey@0"
I20260810 01:51:20.682737 8350 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 01:51:20.686095 8371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:20.687749 8350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 01:51:20.687960 8350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "0994ffb71f304afb8f52cc2a30e3cdd3"
format_stamp: "Formatted at 2026-08-10 01:51:20 on dist-test-slave-kvfs"
server_key: "6163af4848b55a0d39348cd03bb8f33c"
server_key_iv: "4ba06952af8689cb86bec0c21fe94182"
server_key_version: "encryptionkey@0"
I20260810 01:51:20.688177 8350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:20.705487 8350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:20.709494 8350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:20.709795 8350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:20.718912 8350 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36731
I20260810 01:51:20.718932 8454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36731 every 8 connection(s)
I20260810 01:51:20.720257 8350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:51:20.723903 8455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:20.727485 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 8350
I20260810 01:51:20.727665 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:51:20.728171 9526 external_mini_cluster.cc:1428] Setting key 4b498562629f7027131ea6fa1192d916
I20260810 01:51:20.732159 8455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0994ffb71f304afb8f52cc2a30e3cdd3: Bootstrap starting.
I20260810 01:51:20.735064 8455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0994ffb71f304afb8f52cc2a30e3cdd3: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:20.736088 8455 log.cc:824] T 00000000000000000000000000000000 P 0994ffb71f304afb8f52cc2a30e3cdd3: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:20.738513 8455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0994ffb71f304afb8f52cc2a30e3cdd3: No bootstrap required, opened a new log
Aug 10 01:51:20 dist-test-slave-kvfs krb5kdc[8324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 01:51:20.742849 8455 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0994ffb71f304afb8f52cc2a30e3cdd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0994ffb71f304afb8f52cc2a30e3cdd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36731 } }
I20260810 01:51:20.743193 8455 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0994ffb71f304afb8f52cc2a30e3cdd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:20.743287 8455 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0994ffb71f304afb8f52cc2a30e3cdd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0994ffb71f304afb8f52cc2a30e3cdd3, State: Initialized, Role: FOLLOWER
I20260810 01:51:20.743902 8455 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0994ffb71f304afb8f52cc2a30e3cdd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0994ffb71f304afb8f52cc2a30e3cdd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36731 } }
I20260810 01:51:20.744124 8455 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0994ffb71f304afb8f52cc2a30e3cdd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:51:20.744251 8455 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0994ffb71f304afb8f52cc2a30e3cdd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:51:20.744451 8455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0994ffb71f304afb8f52cc2a30e3cdd3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:20.745595 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:20.730153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39339 (local address 127.0.0.1:36731)
0810 01:51:20.730731 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:20.730750 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:20.730797 (+ 47us) server_negotiation.cc:407] Connection header received
0810 01:51:20.731560 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:20.731582 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:20.732078 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:20.732548 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:20.733535 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:20.734680 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:20.735715 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:20.736381 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:20.736573 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:20.739590 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:20.739597 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:20.739614 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:20.739647 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:20.741720 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:20.742333 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:20.742339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:20.742345 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:20.742493 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:20.742881 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:20.742884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:20.742885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:20.743537 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:20.743805 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:20.744205 (+ 400us) server_negotiation.cc:299] Negotiation successful
0810 01:51:20.744502 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":175,"threads_started":1}
I20260810 01:51:20.745738 8455 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0994ffb71f304afb8f52cc2a30e3cdd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0994ffb71f304afb8f52cc2a30e3cdd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36731 } }
I20260810 01:51:20.746219 8455 leader_election.cc:302] T 00000000000000000000000000000000 P 0994ffb71f304afb8f52cc2a30e3cdd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0994ffb71f304afb8f52cc2a30e3cdd3; no voters:
I20260810 01:51:20.746678 8455 leader_election.cc:288] T 00000000000000000000000000000000 P 0994ffb71f304afb8f52cc2a30e3cdd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:51:20.746798 8464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0994ffb71f304afb8f52cc2a30e3cdd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:20.747013 8464 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0994ffb71f304afb8f52cc2a30e3cdd3 [term 1 LEADER]: Becoming Leader. State: Replica: 0994ffb71f304afb8f52cc2a30e3cdd3, State: Running, Role: LEADER
I20260810 01:51:20.747397 8464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0994ffb71f304afb8f52cc2a30e3cdd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0994ffb71f304afb8f52cc2a30e3cdd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36731 } }
I20260810 01:51:20.748431 8455 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0994ffb71f304afb8f52cc2a30e3cdd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:20.748976 8467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0994ffb71f304afb8f52cc2a30e3cdd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0994ffb71f304afb8f52cc2a30e3cdd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0994ffb71f304afb8f52cc2a30e3cdd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36731 } } }
I20260810 01:51:20.749279 8467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0994ffb71f304afb8f52cc2a30e3cdd3 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:20.749820 8472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:51:20.750000 9526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 01:51:20.748983 8464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0994ffb71f304afb8f52cc2a30e3cdd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0994ffb71f304afb8f52cc2a30e3cdd3. Latest consensus state: current_term: 1 leader_uuid: "0994ffb71f304afb8f52cc2a30e3cdd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0994ffb71f304afb8f52cc2a30e3cdd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36731 } } }
I20260810 01:51:20.750711 8464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0994ffb71f304afb8f52cc2a30e3cdd3 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:20.755319 8472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:51:20.763716 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:20.753142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56485 (local address 127.0.0.1:36731)
0810 01:51:20.753285 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:20.753289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:20.753553 (+ 264us) server_negotiation.cc:407] Connection header received
0810 01:51:20.753706 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:20.753709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:20.753798 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:20.753906 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:20.754875 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:20.755720 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:20.756940 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:20.757363 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:20.757489 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:20.759572 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:20.759576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:20.759578 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:20.759605 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:20.761409 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:20.762119 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:20.762124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:20.762126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:20.762254 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:20.762665 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:20.762668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:20.762670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:20.762904 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:20.763077 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:20.763388 (+ 311us) server_negotiation.cc:299] Negotiation successful
0810 01:51:20.763501 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 01:51:20.765110 8472 catalog_manager.cc:1347] Generated new cluster ID: b15eb7c4f1a2415a953cdaa17e143722
I20260810 01:51:20.765185 8472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:51:20.777729 8472 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:51:20.778914 8472 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:51:20.793337 8472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0994ffb71f304afb8f52cc2a30e3cdd3: Generated new TSK 0
I20260810 01:51:20.794592 8472 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:51:20.808566 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 8350
W20260810 01:51:20.827486 8322 connection.cc:535] client connection to 127.0.0.1:36731 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36731): Connection reset by peer (error 104)
2026-08-10T01:51:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1787 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:51:20 dist-test-slave-kvfs krb5kdc[8502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:51:20 dist-test-slave-kvfs krb5kdc[8502](info): set up 2 sockets
Aug 10 01:51:20 dist-test-slave-kvfs krb5kdc[8502](info): commencing operation
krb5kdc: starting...
W20260810 01:51:22.130264 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.287s user 0.003s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:51:22 dist-test-slave-kvfs krb5kdc[8502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:51:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:51:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 01:51:22.287191 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32775
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:22.427806 8529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:22.428259 8529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:22.428344 8529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:22.432873 8529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:22.432993 8529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:22.433034 8529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:22.433073 8529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:22.433112 8529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:22.439174 8529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32775
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:22.440662 8529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:22.441984 8529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:22.452284 8537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:22.452410 8529 server_base.cc:1034] running on GCE node
W20260810 01:51:22.452329 8540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:22.452284 8538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:22.453265 8529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:22.454594 8529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:22.455819 8529 hybrid_clock.cc:648] HybridClock initialized: now 1786326682455796 us; error 35 us; skew 500 ppm
Aug 10 01:51:22 dist-test-slave-kvfs krb5kdc[8502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:22.463425 8529 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 01:51:22.464740 8529 webserver.cc:460] Webserver started at http://127.0.0.1:46207/ using document root <none> and password file <none>
I20260810 01:51:22.465478 8529 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:22.465582 8529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:22.465875 8529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:22.468555 8529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "e631009eeec44a22bdb7acae278876b7"
format_stamp: "Formatted at 2026-08-10 01:51:22 on dist-test-slave-kvfs"
server_key: "c6df7435d25a46ff5e789968273081c4"
server_key_iv: "17f84634fe6efc258c6c1e1e9c645a5e"
server_key_version: "encryptionkey@0"
I20260810 01:51:22.469233 8529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "e631009eeec44a22bdb7acae278876b7"
format_stamp: "Formatted at 2026-08-10 01:51:22 on dist-test-slave-kvfs"
server_key: "c6df7435d25a46ff5e789968273081c4"
server_key_iv: "17f84634fe6efc258c6c1e1e9c645a5e"
server_key_version: "encryptionkey@0"
I20260810 01:51:22.473827 8529 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 01:51:22.476830 8553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:22.478444 8529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:51:22.478621 8529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "e631009eeec44a22bdb7acae278876b7"
format_stamp: "Formatted at 2026-08-10 01:51:22 on dist-test-slave-kvfs"
server_key: "c6df7435d25a46ff5e789968273081c4"
server_key_iv: "17f84634fe6efc258c6c1e1e9c645a5e"
server_key_version: "encryptionkey@0"
I20260810 01:51:22.478770 8529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:22.507113 8529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:22.511420 8529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:22.511682 8529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:22.521055 8529 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32775
I20260810 01:51:22.521101 8627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32775 every 8 connection(s)
I20260810 01:51:22.522279 8529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:51:22.525904 8630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:22.528309 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 8529
I20260810 01:51:22.528501 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:51:22.528855 9526 external_mini_cluster.cc:1428] Setting key ecf55e1ff8706cd57452b3420d1aabee
I20260810 01:51:22.534591 8630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e631009eeec44a22bdb7acae278876b7: Bootstrap starting.
I20260810 01:51:22.538129 8630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e631009eeec44a22bdb7acae278876b7: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:22.539232 8630 log.cc:824] T 00000000000000000000000000000000 P e631009eeec44a22bdb7acae278876b7: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:22.541440 8634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:22.530647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56009 (local address 127.0.0.1:32775)
0810 01:51:22.531167 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:22.531189 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:22.531250 (+ 61us) server_negotiation.cc:407] Connection header received
0810 01:51:22.532285 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:22.532315 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:22.532853 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:22.533316 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:22.533857 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:22.535459 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:22.536710 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:22.537546 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:22.537723 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:22.538813 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:22.538819 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:22.538839 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:22.539487 (+ 648us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:22.539577 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:22.539612 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:22.539776 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:22.540100 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 01:51:22.540263 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":140,"threads_started":1}
I20260810 01:51:22.542274 8630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e631009eeec44a22bdb7acae278876b7: No bootstrap required, opened a new log
I20260810 01:51:22.545034 9526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 01:51:22.547119 8630 raft_consensus.cc:348] T 00000000000000000000000000000000 P e631009eeec44a22bdb7acae278876b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e631009eeec44a22bdb7acae278876b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32775 } }
I20260810 01:51:22.547497 8630 raft_consensus.cc:374] T 00000000000000000000000000000000 P e631009eeec44a22bdb7acae278876b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:22.547566 8630 raft_consensus.cc:729] T 00000000000000000000000000000000 P e631009eeec44a22bdb7acae278876b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e631009eeec44a22bdb7acae278876b7, State: Initialized, Role: FOLLOWER
I20260810 01:51:22.548269 8630 consensus_queue.cc:260] T 00000000000000000000000000000000 P e631009eeec44a22bdb7acae278876b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e631009eeec44a22bdb7acae278876b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32775 } }
I20260810 01:51:22.548476 8630 raft_consensus.cc:388] T 00000000000000000000000000000000 P e631009eeec44a22bdb7acae278876b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:51:22.548559 8630 raft_consensus.cc:482] T 00000000000000000000000000000000 P e631009eeec44a22bdb7acae278876b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:51:22.548699 8630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e631009eeec44a22bdb7acae278876b7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:22.550170 8630 raft_consensus.cc:504] T 00000000000000000000000000000000 P e631009eeec44a22bdb7acae278876b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e631009eeec44a22bdb7acae278876b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32775 } }
I20260810 01:51:22.550654 8630 leader_election.cc:302] T 00000000000000000000000000000000 P e631009eeec44a22bdb7acae278876b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e631009eeec44a22bdb7acae278876b7; no voters:
I20260810 01:51:22.551126 8630 leader_election.cc:288] T 00000000000000000000000000000000 P e631009eeec44a22bdb7acae278876b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:51:22.551234 8647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e631009eeec44a22bdb7acae278876b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:22.551438 8647 raft_consensus.cc:686] T 00000000000000000000000000000000 P e631009eeec44a22bdb7acae278876b7 [term 1 LEADER]: Becoming Leader. State: Replica: e631009eeec44a22bdb7acae278876b7, State: Running, Role: LEADER
I20260810 01:51:22.551803 8647 consensus_queue.cc:237] T 00000000000000000000000000000000 P e631009eeec44a22bdb7acae278876b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e631009eeec44a22bdb7acae278876b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32775 } }
I20260810 01:51:22.552814 8634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:22.547502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54319 (local address 127.0.0.1:32775)
0810 01:51:22.547713 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:22.547718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:22.547976 (+ 258us) server_negotiation.cc:407] Connection header received
0810 01:51:22.548124 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:22.548126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:22.548177 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:22.548250 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:22.548618 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:22.549318 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:22.550526 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:22.550780 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:22.550915 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:22.552291 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:22.552295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:22.552298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:22.552313 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:22.552352 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:22.552364 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:22.552472 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:22.552598 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 01:51:22.552638 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260810 01:51:22.552889 8630 sys_catalog.cc:564] T 00000000000000000000000000000000 P e631009eeec44a22bdb7acae278876b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:22.553668 8647 sys_catalog.cc:455] T 00000000000000000000000000000000 P e631009eeec44a22bdb7acae278876b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e631009eeec44a22bdb7acae278876b7. Latest consensus state: current_term: 1 leader_uuid: "e631009eeec44a22bdb7acae278876b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e631009eeec44a22bdb7acae278876b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32775 } } }
I20260810 01:51:22.553822 8647 sys_catalog.cc:458] T 00000000000000000000000000000000 P e631009eeec44a22bdb7acae278876b7 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:22.554848 8658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:51:22.554788 8649 sys_catalog.cc:455] T 00000000000000000000000000000000 P e631009eeec44a22bdb7acae278876b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e631009eeec44a22bdb7acae278876b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e631009eeec44a22bdb7acae278876b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32775 } } }
I20260810 01:51:22.554916 8649 sys_catalog.cc:458] T 00000000000000000000000000000000 P e631009eeec44a22bdb7acae278876b7 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:22.558585 8658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:51:22.565522 8658 catalog_manager.cc:1347] Generated new cluster ID: af78e00fa87c4c12bf330df705522888
I20260810 01:51:22.565634 8658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:51:22.573714 8658 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:51:22.574904 8658 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:51:22.592473 8658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e631009eeec44a22bdb7acae278876b7: Generated new TSK 0
I20260810 01:51:22.593674 8658 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:51:22.603055 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 8529
W20260810 01:51:22.621141 8495 connection.cc:535] client connection to 127.0.0.1:32775 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32775): Connection reset by peer (error 104)
2026-08-10T01:51:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1793 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:51:22 dist-test-slave-kvfs krb5kdc[8678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:51:22 dist-test-slave-kvfs krb5kdc[8678](info): set up 2 sockets
Aug 10 01:51:22 dist-test-slave-kvfs krb5kdc[8678](info): commencing operation
krb5kdc: starting...
W20260810 01:51:23.909454 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.273s user 0.001s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:51:24 dist-test-slave-kvfs krb5kdc[8678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:51:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:51:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 01:51:24.068388 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43885
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:24.211256 8699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:24.211664 8699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:24.211715 8699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:24.215971 8699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:24.216081 8699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:24.216104 8699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:24.216121 8699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:24.216137 8699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:24.222133 8699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43885
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:24.223678 8699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:24.225054 8699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:24.236100 8710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:24.236104 8712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:24.236100 8709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:24.236250 8699 server_base.cc:1034] running on GCE node
I20260810 01:51:24.237311 8699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:24.238865 8699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:24.240095 8699 hybrid_clock.cc:648] HybridClock initialized: now 1786326684240072 us; error 36 us; skew 500 ppm
Aug 10 01:51:24 dist-test-slave-kvfs krb5kdc[8678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:24.243835 8699 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 01:51:24.245296 8699 webserver.cc:460] Webserver started at http://127.0.0.1:45033/ using document root <none> and password file <none>
I20260810 01:51:24.246073 8699 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:24.246155 8699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:24.246500 8699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:24.249321 8699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "540b1c8c4b9844eda64d20075b9c5249"
format_stamp: "Formatted at 2026-08-10 01:51:24 on dist-test-slave-kvfs"
server_key: "c653d7bda556e48397e952eac90b98ee"
server_key_iv: "d23d811e3748467493799fb2d19c200e"
server_key_version: "encryptionkey@0"
I20260810 01:51:24.250047 8699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "540b1c8c4b9844eda64d20075b9c5249"
format_stamp: "Formatted at 2026-08-10 01:51:24 on dist-test-slave-kvfs"
server_key: "c653d7bda556e48397e952eac90b98ee"
server_key_iv: "d23d811e3748467493799fb2d19c200e"
server_key_version: "encryptionkey@0"
I20260810 01:51:24.254980 8699 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 01:51:24.258160 8728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:24.259814 8699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 01:51:24.260005 8699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "540b1c8c4b9844eda64d20075b9c5249"
format_stamp: "Formatted at 2026-08-10 01:51:24 on dist-test-slave-kvfs"
server_key: "c653d7bda556e48397e952eac90b98ee"
server_key_iv: "d23d811e3748467493799fb2d19c200e"
server_key_version: "encryptionkey@0"
I20260810 01:51:24.260233 8699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:24.266363 8699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:24.270314 8699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:24.270545 8699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:24.279248 8699 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43885
I20260810 01:51:24.279276 8817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43885 every 8 connection(s)
I20260810 01:51:24.280536 8699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:51:24.284508 8819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:24.289274 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 8699
I20260810 01:51:24.289434 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:51:24.289821 9526 external_mini_cluster.cc:1428] Setting key ec79fd978f7ccea9bdc378c0e321b2c4
I20260810 01:51:24.292651 8819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 540b1c8c4b9844eda64d20075b9c5249: Bootstrap starting.
I20260810 01:51:24.296217 8819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 540b1c8c4b9844eda64d20075b9c5249: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:24.296933 8824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:24.291720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58035 (local address 127.0.0.1:43885)
0810 01:51:24.292330 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:24.292351 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:24.292406 (+ 55us) server_negotiation.cc:407] Connection header received
0810 01:51:24.293405 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:24.293438 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:24.293878 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:24.294363 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:24.295212 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:24.295225 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:24.295241 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:24.295254 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:24.295330 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:24.295359 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:24.295475 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:24.295573 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 01:51:24.295755 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":239,"threads_started":1}
I20260810 01:51:24.297299 8819 log.cc:824] T 00000000000000000000000000000000 P 540b1c8c4b9844eda64d20075b9c5249: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:24.300004 8819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 540b1c8c4b9844eda64d20075b9c5249: No bootstrap required, opened a new log
I20260810 01:51:24.300416 9526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 01:51:24.305140 8824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:24.303052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37749 (local address 127.0.0.1:43885)
0810 01:51:24.303259 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:24.303263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:24.303417 (+ 154us) server_negotiation.cc:407] Connection header received
0810 01:51:24.303570 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:24.303572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:24.303609 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:24.303685 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:24.304648 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:24.304651 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:24.304654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:24.304665 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:24.304698 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:24.304709 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:24.304801 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:24.304905 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 01:51:24.304947 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260810 01:51:24.305325 8819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 540b1c8c4b9844eda64d20075b9c5249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540b1c8c4b9844eda64d20075b9c5249" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43885 } }
I20260810 01:51:24.305706 8819 raft_consensus.cc:374] T 00000000000000000000000000000000 P 540b1c8c4b9844eda64d20075b9c5249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:24.305770 8819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 540b1c8c4b9844eda64d20075b9c5249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 540b1c8c4b9844eda64d20075b9c5249, State: Initialized, Role: FOLLOWER
I20260810 01:51:24.306305 8819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 540b1c8c4b9844eda64d20075b9c5249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540b1c8c4b9844eda64d20075b9c5249" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43885 } }
I20260810 01:51:24.306485 8819 raft_consensus.cc:388] T 00000000000000000000000000000000 P 540b1c8c4b9844eda64d20075b9c5249 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:51:24.306561 8819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 540b1c8c4b9844eda64d20075b9c5249 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:51:24.306676 8819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 540b1c8c4b9844eda64d20075b9c5249 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:24.307955 8819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 540b1c8c4b9844eda64d20075b9c5249 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540b1c8c4b9844eda64d20075b9c5249" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43885 } }
I20260810 01:51:24.308427 8819 leader_election.cc:302] T 00000000000000000000000000000000 P 540b1c8c4b9844eda64d20075b9c5249 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 540b1c8c4b9844eda64d20075b9c5249; no voters:
I20260810 01:51:24.308846 8819 leader_election.cc:288] T 00000000000000000000000000000000 P 540b1c8c4b9844eda64d20075b9c5249 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:51:24.309008 8833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 540b1c8c4b9844eda64d20075b9c5249 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:24.309253 8833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 540b1c8c4b9844eda64d20075b9c5249 [term 1 LEADER]: Becoming Leader. State: Replica: 540b1c8c4b9844eda64d20075b9c5249, State: Running, Role: LEADER
I20260810 01:51:24.309711 8833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 540b1c8c4b9844eda64d20075b9c5249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540b1c8c4b9844eda64d20075b9c5249" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43885 } }
I20260810 01:51:24.310520 8819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 540b1c8c4b9844eda64d20075b9c5249 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:24.311769 8835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540b1c8c4b9844eda64d20075b9c5249 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 540b1c8c4b9844eda64d20075b9c5249. Latest consensus state: current_term: 1 leader_uuid: "540b1c8c4b9844eda64d20075b9c5249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540b1c8c4b9844eda64d20075b9c5249" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43885 } } }
I20260810 01:51:24.311915 8835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540b1c8c4b9844eda64d20075b9c5249 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:24.312288 8834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540b1c8c4b9844eda64d20075b9c5249 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "540b1c8c4b9844eda64d20075b9c5249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540b1c8c4b9844eda64d20075b9c5249" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43885 } } }
I20260810 01:51:24.312386 8834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540b1c8c4b9844eda64d20075b9c5249 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:24.312400 8851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:51:24.316396 8851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:51:24.323700 8851 catalog_manager.cc:1347] Generated new cluster ID: 6ff0a65bc6924d7785eda68e4dab847e
I20260810 01:51:24.323863 8851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:51:24.339325 8851 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:51:24.339866 8851 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:51:24.350167 8851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 540b1c8c4b9844eda64d20075b9c5249: Generated new TSK 0
I20260810 01:51:24.352157 8851 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:51:24.364720 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 8699
2026-08-10T01:51:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1774 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:51:24 dist-test-slave-kvfs krb5kdc[8872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:51:24 dist-test-slave-kvfs krb5kdc[8872](info): set up 2 sockets
Aug 10 01:51:24 dist-test-slave-kvfs krb5kdc[8872](info): commencing operation
krb5kdc: starting...
W20260810 01:51:25.668919 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.258s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:51:25 dist-test-slave-kvfs krb5kdc[8872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:51:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:51:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 01:51:25.824357 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35007
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:25.971786 8892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:25.972278 8892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:25.972374 8892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:25.976747 8892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:25.976859 8892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:25.976899 8892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:25.976959 8892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:25.976998 8892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:25.983244 8892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:25.984910 8892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:25.986323 8892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:25.997251 8897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:25.997251 8905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:25.997289 8899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:25.997702 8892 server_base.cc:1034] running on GCE node
I20260810 01:51:25.998483 8892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:26.000090 8892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:26.001305 8892 hybrid_clock.cc:648] HybridClock initialized: now 1786326686001282 us; error 39 us; skew 500 ppm
Aug 10 01:51:26 dist-test-slave-kvfs krb5kdc[8872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:26.005026 8892 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 01:51:26.006286 8892 webserver.cc:460] Webserver started at http://127.0.0.1:43781/ using document root <none> and password file <none>
I20260810 01:51:26.007035 8892 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:26.007149 8892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:26.007480 8892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:26.010329 8892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "899796ad04374b299eb108823ec3ec5e"
format_stamp: "Formatted at 2026-08-10 01:51:26 on dist-test-slave-kvfs"
server_key: "c5123ed62bec20d9e24b5ec59bcd6972"
server_key_iv: "491efe1c25fe6003e4a8ddb9dbd4a92d"
server_key_version: "encryptionkey@0"
I20260810 01:51:26.011101 8892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "899796ad04374b299eb108823ec3ec5e"
format_stamp: "Formatted at 2026-08-10 01:51:26 on dist-test-slave-kvfs"
server_key: "c5123ed62bec20d9e24b5ec59bcd6972"
server_key_iv: "491efe1c25fe6003e4a8ddb9dbd4a92d"
server_key_version: "encryptionkey@0"
I20260810 01:51:26.016314 8892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260810 01:51:26.019817 8916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:26.021592 8892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:51:26.021781 8892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "899796ad04374b299eb108823ec3ec5e"
format_stamp: "Formatted at 2026-08-10 01:51:26 on dist-test-slave-kvfs"
server_key: "c5123ed62bec20d9e24b5ec59bcd6972"
server_key_iv: "491efe1c25fe6003e4a8ddb9dbd4a92d"
server_key_version: "encryptionkey@0"
I20260810 01:51:26.021944 8892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:26.042491 8892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:26.046741 8892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:26.047075 8892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:26.056237 8892 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35007
I20260810 01:51:26.056315 8999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35007 every 8 connection(s)
I20260810 01:51:26.057606 8892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:51:26.059229 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 8892
I20260810 01:51:26.059415 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:51:26.059829 9526 external_mini_cluster.cc:1428] Setting key ef3814fc01c60af3c86174efb1e74358
I20260810 01:51:26.062321 9004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:26.070674 9004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 899796ad04374b299eb108823ec3ec5e: Bootstrap starting.
Aug 10 01:51:26 dist-test-slave-kvfs krb5kdc[8872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 01:51:26.074409 9004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 899796ad04374b299eb108823ec3ec5e: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:26.075642 9004 log.cc:824] T 00000000000000000000000000000000 P 899796ad04374b299eb108823ec3ec5e: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:26.077821 9012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:26.061702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46871 (local address 127.0.0.1:35007)
0810 01:51:26.062340 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:26.062366 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:26.062432 (+ 66us) server_negotiation.cc:407] Connection header received
0810 01:51:26.063338 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:26.063360 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:26.063827 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:26.064355 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:26.065391 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:26.066581 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:26.067711 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:26.068353 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:26.068548 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:26.071827 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:26.071854 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:26.071871 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:26.071909 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:26.074124 (+ 2215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:26.074737 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:26.074744 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:26.074750 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:26.074892 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:26.075242 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:26.075244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:26.075246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:26.075861 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:26.076108 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:26.076491 (+ 383us) server_negotiation.cc:299] Negotiation successful
0810 01:51:26.076873 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":214,"threads_started":1}
I20260810 01:51:26.078372 9004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 899796ad04374b299eb108823ec3ec5e: No bootstrap required, opened a new log
I20260810 01:51:26.081235 9526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 01:51:26.082979 9004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 899796ad04374b299eb108823ec3ec5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899796ad04374b299eb108823ec3ec5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35007 } }
I20260810 01:51:26.083292 9004 raft_consensus.cc:374] T 00000000000000000000000000000000 P 899796ad04374b299eb108823ec3ec5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:26.083380 9004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 899796ad04374b299eb108823ec3ec5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 899796ad04374b299eb108823ec3ec5e, State: Initialized, Role: FOLLOWER
I20260810 01:51:26.084108 9004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 899796ad04374b299eb108823ec3ec5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899796ad04374b299eb108823ec3ec5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35007 } }
I20260810 01:51:26.084312 9004 raft_consensus.cc:388] T 00000000000000000000000000000000 P 899796ad04374b299eb108823ec3ec5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:51:26.084404 9004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 899796ad04374b299eb108823ec3ec5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:51:26.084532 9004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 899796ad04374b299eb108823ec3ec5e [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:26.085923 9004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 899796ad04374b299eb108823ec3ec5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899796ad04374b299eb108823ec3ec5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35007 } }
I20260810 01:51:26.086414 9004 leader_election.cc:302] T 00000000000000000000000000000000 P 899796ad04374b299eb108823ec3ec5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 899796ad04374b299eb108823ec3ec5e; no voters:
I20260810 01:51:26.086839 9004 leader_election.cc:288] T 00000000000000000000000000000000 P 899796ad04374b299eb108823ec3ec5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:51:26.086969 9019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 899796ad04374b299eb108823ec3ec5e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:26.087251 9019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 899796ad04374b299eb108823ec3ec5e [term 1 LEADER]: Becoming Leader. State: Replica: 899796ad04374b299eb108823ec3ec5e, State: Running, Role: LEADER
I20260810 01:51:26.087721 9019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 899796ad04374b299eb108823ec3ec5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899796ad04374b299eb108823ec3ec5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35007 } }
I20260810 01:51:26.089046 9004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 899796ad04374b299eb108823ec3ec5e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:26.089802 9021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 899796ad04374b299eb108823ec3ec5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 899796ad04374b299eb108823ec3ec5e. Latest consensus state: current_term: 1 leader_uuid: "899796ad04374b299eb108823ec3ec5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899796ad04374b299eb108823ec3ec5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35007 } } }
I20260810 01:51:26.089941 9021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 899796ad04374b299eb108823ec3ec5e [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:26.090325 9020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 899796ad04374b299eb108823ec3ec5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "899796ad04374b299eb108823ec3ec5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899796ad04374b299eb108823ec3ec5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35007 } } }
I20260810 01:51:26.090512 9020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 899796ad04374b299eb108823ec3ec5e [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:26.091691 9025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:51:26.094827 9012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:26.083858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48885 (local address 127.0.0.1:35007)
0810 01:51:26.084037 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:26.084042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:26.084217 (+ 175us) server_negotiation.cc:407] Connection header received
0810 01:51:26.084372 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:26.084376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:26.084442 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:26.084535 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:26.085327 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:26.086190 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:26.087332 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:26.087714 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:26.087835 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:26.089816 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:26.089820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:26.089823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:26.089849 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:26.092355 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:26.092833 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:26.092837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:26.092839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:26.092937 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:26.093234 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:26.093237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:26.093239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:26.093514 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:26.093649 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:26.094507 (+ 858us) server_negotiation.cc:299] Negotiation successful
0810 01:51:26.094642 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 01:51:26.095690 9025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:51:26.102741 9025 catalog_manager.cc:1347] Generated new cluster ID: c7a5ca494a05441090e5d1645d414d0b
I20260810 01:51:26.102833 9025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:51:26.110868 9025 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:51:26.112388 9025 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:51:26.132015 9025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 899796ad04374b299eb108823ec3ec5e: Generated new TSK 0
I20260810 01:51:26.133258 9025 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:51:26.142109 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 8892
2026-08-10T01:51:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1761 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:51:26 dist-test-slave-kvfs krb5kdc[9050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:51:26 dist-test-slave-kvfs krb5kdc[9050](info): set up 2 sockets
Aug 10 01:51:26 dist-test-slave-kvfs krb5kdc[9050](info): commencing operation
krb5kdc: starting...
W20260810 01:51:27.474340 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.302s user 0.005s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:51:27 dist-test-slave-kvfs krb5kdc[9050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:51:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:51:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 01:51:27.640786 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43519
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:27.781198 9066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:27.781571 9066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:27.781652 9066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:27.785833 9066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:27.785907 9066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:27.785926 9066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:27.785943 9066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:27.785959 9066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:27.791764 9066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:27.793299 9066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:27.794652 9066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:27.805002 9079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:27.804994 9075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:27.804994 9076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:27.805467 9066 server_base.cc:1034] running on GCE node
I20260810 01:51:27.806145 9066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:27.807500 9066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:27.808720 9066 hybrid_clock.cc:648] HybridClock initialized: now 1786326687808651 us; error 60 us; skew 500 ppm
Aug 10 01:51:27 dist-test-slave-kvfs krb5kdc[9050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:27.817134 9066 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 01:51:27.818466 9066 webserver.cc:460] Webserver started at http://127.0.0.1:43645/ using document root <none> and password file <none>
I20260810 01:51:27.819231 9066 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:27.819337 9066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:27.819640 9066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:27.822414 9066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "56cb4fc86a0f4bf1b9c53ae084aedc49"
format_stamp: "Formatted at 2026-08-10 01:51:27 on dist-test-slave-kvfs"
server_key: "baf711c669627e00697c05a1cec3180a"
server_key_iv: "8c3f0c2e1388307297e462055ea1bde8"
server_key_version: "encryptionkey@0"
I20260810 01:51:27.823120 9066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "56cb4fc86a0f4bf1b9c53ae084aedc49"
format_stamp: "Formatted at 2026-08-10 01:51:27 on dist-test-slave-kvfs"
server_key: "baf711c669627e00697c05a1cec3180a"
server_key_iv: "8c3f0c2e1388307297e462055ea1bde8"
server_key_version: "encryptionkey@0"
I20260810 01:51:27.827752 9066 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260810 01:51:27.831071 9092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:27.832819 9066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:51:27.833016 9066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "56cb4fc86a0f4bf1b9c53ae084aedc49"
format_stamp: "Formatted at 2026-08-10 01:51:27 on dist-test-slave-kvfs"
server_key: "baf711c669627e00697c05a1cec3180a"
server_key_iv: "8c3f0c2e1388307297e462055ea1bde8"
server_key_version: "encryptionkey@0"
I20260810 01:51:27.833195 9066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:27.849450 9066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:27.853410 9066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:27.853659 9066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:27.862798 9066 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43519
I20260810 01:51:27.862833 9170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43519 every 8 connection(s)
I20260810 01:51:27.864077 9066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:51:27.867602 9171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:27.871382 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 9066
I20260810 01:51:27.871582 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:51:27.871927 9526 external_mini_cluster.cc:1428] Setting key 90dd3bec4348542a43562f8be4e93220
I20260810 01:51:27.875567 9171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56cb4fc86a0f4bf1b9c53ae084aedc49: Bootstrap starting.
I20260810 01:51:27.878453 9171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56cb4fc86a0f4bf1b9c53ae084aedc49: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:27.879496 9171 log.cc:824] T 00000000000000000000000000000000 P 56cb4fc86a0f4bf1b9c53ae084aedc49: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:27.882359 9171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56cb4fc86a0f4bf1b9c53ae084aedc49: No bootstrap required, opened a new log
I20260810 01:51:27.883491 9175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:27.873880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44671 (local address 127.0.0.1:43519)
0810 01:51:27.874470 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:27.874489 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:27.874535 (+ 46us) server_negotiation.cc:407] Connection header received
0810 01:51:27.875247 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:27.875268 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:27.875760 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:27.876200 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:27.876703 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:27.877903 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:27.879033 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:27.879811 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:27.880071 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:27.881108 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:27.881116 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:27.881135 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:27.881739 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:27.881818 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:27.881851 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:27.881962 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:27.882073 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 01:51:27.882325 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":196,"threads_started":1}
I20260810 01:51:27.887071 9526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 01:51:27.887310 9171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56cb4fc86a0f4bf1b9c53ae084aedc49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cb4fc86a0f4bf1b9c53ae084aedc49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43519 } }
I20260810 01:51:27.887602 9171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56cb4fc86a0f4bf1b9c53ae084aedc49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:27.887678 9171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56cb4fc86a0f4bf1b9c53ae084aedc49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56cb4fc86a0f4bf1b9c53ae084aedc49, State: Initialized, Role: FOLLOWER
I20260810 01:51:27.888306 9171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56cb4fc86a0f4bf1b9c53ae084aedc49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cb4fc86a0f4bf1b9c53ae084aedc49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43519 } }
I20260810 01:51:27.888459 9171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56cb4fc86a0f4bf1b9c53ae084aedc49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:51:27.888548 9171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56cb4fc86a0f4bf1b9c53ae084aedc49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:51:27.888684 9171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56cb4fc86a0f4bf1b9c53ae084aedc49 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:27.889886 9171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56cb4fc86a0f4bf1b9c53ae084aedc49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cb4fc86a0f4bf1b9c53ae084aedc49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43519 } }
I20260810 01:51:27.890372 9171 leader_election.cc:302] T 00000000000000000000000000000000 P 56cb4fc86a0f4bf1b9c53ae084aedc49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56cb4fc86a0f4bf1b9c53ae084aedc49; no voters:
I20260810 01:51:27.890810 9171 leader_election.cc:288] T 00000000000000000000000000000000 P 56cb4fc86a0f4bf1b9c53ae084aedc49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:51:27.890924 9186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56cb4fc86a0f4bf1b9c53ae084aedc49 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:27.891158 9186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56cb4fc86a0f4bf1b9c53ae084aedc49 [term 1 LEADER]: Becoming Leader. State: Replica: 56cb4fc86a0f4bf1b9c53ae084aedc49, State: Running, Role: LEADER
I20260810 01:51:27.891652 9186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56cb4fc86a0f4bf1b9c53ae084aedc49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cb4fc86a0f4bf1b9c53ae084aedc49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43519 } }
I20260810 01:51:27.892604 9171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56cb4fc86a0f4bf1b9c53ae084aedc49 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:27.893800 9187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56cb4fc86a0f4bf1b9c53ae084aedc49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56cb4fc86a0f4bf1b9c53ae084aedc49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cb4fc86a0f4bf1b9c53ae084aedc49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43519 } } }
I20260810 01:51:27.893961 9187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56cb4fc86a0f4bf1b9c53ae084aedc49 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:27.894671 9200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:51:27.894829 9188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56cb4fc86a0f4bf1b9c53ae084aedc49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56cb4fc86a0f4bf1b9c53ae084aedc49. Latest consensus state: current_term: 1 leader_uuid: "56cb4fc86a0f4bf1b9c53ae084aedc49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56cb4fc86a0f4bf1b9c53ae084aedc49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43519 } } }
I20260810 01:51:27.895007 9188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56cb4fc86a0f4bf1b9c53ae084aedc49 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:27.896167 9175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:27.890210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48129 (local address 127.0.0.1:43519)
0810 01:51:27.890375 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:27.890381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:27.890601 (+ 220us) server_negotiation.cc:407] Connection header received
0810 01:51:27.890793 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:27.890798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:27.890861 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:27.890967 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:27.891494 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:27.892340 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:27.893560 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:27.893983 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:27.894106 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:27.895347 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:27.895351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:27.895354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:27.895370 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:27.895404 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:27.895418 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:27.895491 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:27.895885 (+ 394us) server_negotiation.cc:299] Negotiation successful
0810 01:51:27.895984 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 01:51:27.899155 9200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:51:27.906085 9200 catalog_manager.cc:1347] Generated new cluster ID: 753eb9e956fb4d7ca827f26f3723d517
I20260810 01:51:27.906255 9200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:51:27.925652 9200 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:51:27.927402 9200 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:51:27.941087 9200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56cb4fc86a0f4bf1b9c53ae084aedc49: Generated new TSK 0
I20260810 01:51:27.942297 9200 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:51:27.949141 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 9066
2026-08-10T01:51:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1812 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:51:28 dist-test-slave-kvfs krb5kdc[9216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:51:28 dist-test-slave-kvfs krb5kdc[9216](info): set up 2 sockets
Aug 10 01:51:28 dist-test-slave-kvfs krb5kdc[9216](info): commencing operation
krb5kdc: starting...
W20260810 01:51:29.234165 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.249s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:51:29 dist-test-slave-kvfs krb5kdc[9216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:51:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:51:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 01:51:29.388324 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35419
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:29.525966 9232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:29.526396 9232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:29.526489 9232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:29.530773 9232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:29.530882 9232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:29.530922 9232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:29.530977 9232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:29.531016 9232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:29.536849 9232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:29.538517 9232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:29.540176 9232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:29.550482 9238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:29.550482 9239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:29.550503 9241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:29.551013 9232 server_base.cc:1034] running on GCE node
I20260810 01:51:29.551689 9232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:29.553180 9232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:29.554401 9232 hybrid_clock.cc:648] HybridClock initialized: now 1786326689554369 us; error 36 us; skew 500 ppm
Aug 10 01:51:29 dist-test-slave-kvfs krb5kdc[9216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:29.557806 9232 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 01:51:29.558947 9232 webserver.cc:460] Webserver started at http://127.0.0.1:41585/ using document root <none> and password file <none>
I20260810 01:51:29.559628 9232 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:29.559693 9232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:29.559916 9232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:29.562497 9232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "87556bb65ab740f6bdbfea88708a76ec"
format_stamp: "Formatted at 2026-08-10 01:51:29 on dist-test-slave-kvfs"
server_key: "91e1341a199a2eb39a597cbdc315525f"
server_key_iv: "03a57d4ba76ce90e8c1d082ecbd35312"
server_key_version: "encryptionkey@0"
I20260810 01:51:29.563113 9232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "87556bb65ab740f6bdbfea88708a76ec"
format_stamp: "Formatted at 2026-08-10 01:51:29 on dist-test-slave-kvfs"
server_key: "91e1341a199a2eb39a597cbdc315525f"
server_key_iv: "03a57d4ba76ce90e8c1d082ecbd35312"
server_key_version: "encryptionkey@0"
I20260810 01:51:29.567540 9232 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260810 01:51:29.570609 9251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:29.572216 9232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:51:29.572350 9232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "87556bb65ab740f6bdbfea88708a76ec"
format_stamp: "Formatted at 2026-08-10 01:51:29 on dist-test-slave-kvfs"
server_key: "91e1341a199a2eb39a597cbdc315525f"
server_key_iv: "03a57d4ba76ce90e8c1d082ecbd35312"
server_key_version: "encryptionkey@0"
I20260810 01:51:29.572464 9232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:29.577863 9232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:29.581370 9232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:29.581547 9232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:29.590245 9232 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35419
I20260810 01:51:29.590250 9332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35419 every 8 connection(s)
I20260810 01:51:29.591393 9232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:51:29.594974 9333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:29.598503 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 9232
I20260810 01:51:29.598681 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:51:29.599020 9526 external_mini_cluster.cc:1428] Setting key bbcb1e3033b00499b0735697e93f7875
I20260810 01:51:29.602300 9333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87556bb65ab740f6bdbfea88708a76ec: Bootstrap starting.
I20260810 01:51:29.605252 9338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:29.600784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43617 (local address 127.0.0.1:35419)
0810 01:51:29.601340 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:29.601358 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:29.601402 (+ 44us) server_negotiation.cc:407] Connection header received
0810 01:51:29.602116 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:29.602135 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:29.602524 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:29.602924 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:29.603832 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:29.603837 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:29.603850 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:29.603861 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:29.603924 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:29.603945 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:29.604135 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:29.604217 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 01:51:29.604401 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":175,"threads_started":1}
I20260810 01:51:29.605365 9333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87556bb65ab740f6bdbfea88708a76ec: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:29.606251 9333 log.cc:824] T 00000000000000000000000000000000 P 87556bb65ab740f6bdbfea88708a76ec: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:29.608800 9526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 01:51:29.609251 9333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87556bb65ab740f6bdbfea88708a76ec: No bootstrap required, opened a new log
I20260810 01:51:29.613938 9338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:29.612011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52101 (local address 127.0.0.1:35419)
0810 01:51:29.612186 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:29.612191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:29.612325 (+ 134us) server_negotiation.cc:407] Connection header received
0810 01:51:29.612514 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:29.612517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:29.612561 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:29.612649 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:29.613525 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:29.613529 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:29.613532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:29.613551 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:29.613579 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:29.613589 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:29.613682 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:29.613757 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 01:51:29.613800 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 01:51:29.614336 9333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87556bb65ab740f6bdbfea88708a76ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87556bb65ab740f6bdbfea88708a76ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35419 } }
I20260810 01:51:29.614683 9333 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87556bb65ab740f6bdbfea88708a76ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:29.614748 9333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87556bb65ab740f6bdbfea88708a76ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87556bb65ab740f6bdbfea88708a76ec, State: Initialized, Role: FOLLOWER
I20260810 01:51:29.615294 9333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87556bb65ab740f6bdbfea88708a76ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87556bb65ab740f6bdbfea88708a76ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35419 } }
I20260810 01:51:29.615461 9333 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87556bb65ab740f6bdbfea88708a76ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:51:29.615512 9333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87556bb65ab740f6bdbfea88708a76ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:51:29.615595 9333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87556bb65ab740f6bdbfea88708a76ec [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:29.616694 9333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87556bb65ab740f6bdbfea88708a76ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87556bb65ab740f6bdbfea88708a76ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35419 } }
I20260810 01:51:29.617054 9333 leader_election.cc:302] T 00000000000000000000000000000000 P 87556bb65ab740f6bdbfea88708a76ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87556bb65ab740f6bdbfea88708a76ec; no voters:
I20260810 01:51:29.617357 9333 leader_election.cc:288] T 00000000000000000000000000000000 P 87556bb65ab740f6bdbfea88708a76ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:51:29.617496 9348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87556bb65ab740f6bdbfea88708a76ec [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:29.617739 9348 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87556bb65ab740f6bdbfea88708a76ec [term 1 LEADER]: Becoming Leader. State: Replica: 87556bb65ab740f6bdbfea88708a76ec, State: Running, Role: LEADER
I20260810 01:51:29.618139 9348 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87556bb65ab740f6bdbfea88708a76ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87556bb65ab740f6bdbfea88708a76ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35419 } }
I20260810 01:51:29.618858 9333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87556bb65ab740f6bdbfea88708a76ec [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:29.620306 9349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87556bb65ab740f6bdbfea88708a76ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87556bb65ab740f6bdbfea88708a76ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87556bb65ab740f6bdbfea88708a76ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35419 } } }
I20260810 01:51:29.620446 9349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87556bb65ab740f6bdbfea88708a76ec [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:29.620874 9350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87556bb65ab740f6bdbfea88708a76ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87556bb65ab740f6bdbfea88708a76ec. Latest consensus state: current_term: 1 leader_uuid: "87556bb65ab740f6bdbfea88708a76ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87556bb65ab740f6bdbfea88708a76ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35419 } } }
I20260810 01:51:29.621075 9350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87556bb65ab740f6bdbfea88708a76ec [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:29.620886 9361 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:51:29.624745 9361 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:51:29.631510 9361 catalog_manager.cc:1347] Generated new cluster ID: cdd21c4db4954722bb71cc3b481191fe
I20260810 01:51:29.631598 9361 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:51:29.666392 9361 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:51:29.666857 9361 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:51:29.676810 9361 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87556bb65ab740f6bdbfea88708a76ec: Generated new TSK 0
I20260810 01:51:29.677889 9361 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:51:29.686014 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 9232
2026-08-10T01:51:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1738 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:51:29 dist-test-slave-kvfs krb5kdc[9376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:51:29 dist-test-slave-kvfs krb5kdc[9376](info): set up 2 sockets
Aug 10 01:51:29 dist-test-slave-kvfs krb5kdc[9376](info): commencing operation
krb5kdc: starting...
W20260810 01:51:31.028908 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.305s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:51:31 dist-test-slave-kvfs krb5kdc[9376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:51:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:51:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 01:51:31.182071 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46567
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:31.323416 9400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:31.323839 9400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:31.323932 9400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:31.328544 9400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:31.328679 9400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:31.328727 9400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:31.328794 9400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:31.328836 9400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:31.336675 9400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:31.338290 9400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:31.339766 9400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:31.351295 9406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:31.351297 9408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:31.351297 9405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:31.351434 9400 server_base.cc:1034] running on GCE node
I20260810 01:51:31.352480 9400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:31.353986 9400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:31.355235 9400 hybrid_clock.cc:648] HybridClock initialized: now 1786326691355190 us; error 48 us; skew 500 ppm
Aug 10 01:51:31 dist-test-slave-kvfs krb5kdc[9376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:31.358668 9400 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 01:51:31.359768 9400 webserver.cc:460] Webserver started at http://127.0.0.1:37171/ using document root <none> and password file <none>
I20260810 01:51:31.360502 9400 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:31.360569 9400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:31.360772 9400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:31.363199 9400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "920c769f59d045368940d5adad6cc4a0"
format_stamp: "Formatted at 2026-08-10 01:51:31 on dist-test-slave-kvfs"
server_key: "91b986f430a69d5663967b0699874afd"
server_key_iv: "66d85cee622bde060eb9d67d5610cd17"
server_key_version: "encryptionkey@0"
I20260810 01:51:31.363821 9400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "920c769f59d045368940d5adad6cc4a0"
format_stamp: "Formatted at 2026-08-10 01:51:31 on dist-test-slave-kvfs"
server_key: "91b986f430a69d5663967b0699874afd"
server_key_iv: "66d85cee622bde060eb9d67d5610cd17"
server_key_version: "encryptionkey@0"
I20260810 01:51:31.368455 9400 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 01:51:31.371394 9417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:31.373067 9400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:51:31.373189 9400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "920c769f59d045368940d5adad6cc4a0"
format_stamp: "Formatted at 2026-08-10 01:51:31 on dist-test-slave-kvfs"
server_key: "91b986f430a69d5663967b0699874afd"
server_key_iv: "66d85cee622bde060eb9d67d5610cd17"
server_key_version: "encryptionkey@0"
I20260810 01:51:31.373311 9400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:31.413245 9400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:31.417534 9400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:31.417758 9400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:31.427623 9400 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46567
I20260810 01:51:31.427650 9511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46567 every 8 connection(s)
I20260810 01:51:31.428974 9400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:51:31.432579 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 9400
I20260810 01:51:31.432778 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:51:31.433151 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:31.433297 9526 external_mini_cluster.cc:1428] Setting key bb93acde1a8cb77c49bc512cb3ad60d7
I20260810 01:51:31.442456 9512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 920c769f59d045368940d5adad6cc4a0: Bootstrap starting.
Aug 10 01:51:31 dist-test-slave-kvfs krb5kdc[9376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 01:51:31.446070 9512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 920c769f59d045368940d5adad6cc4a0: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:31.447206 9512 log.cc:824] T 00000000000000000000000000000000 P 920c769f59d045368940d5adad6cc4a0: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:31.450145 9512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 920c769f59d045368940d5adad6cc4a0: No bootstrap required, opened a new log
I20260810 01:51:31.451182 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:31.435391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:55785 (local address 127.0.0.1:46567)
0810 01:51:31.435965 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:31.435984 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:31.436080 (+ 96us) server_negotiation.cc:407] Connection header received
0810 01:51:31.436880 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:31.436917 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:31.437355 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:31.437699 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:31.438830 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:31.439950 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:31.441353 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:31.441903 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:31.442036 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:31.445344 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:31.445350 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:31.445365 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:31.445398 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:31.447726 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:31.448401 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:31.448407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:31.448413 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:31.448532 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:31.448909 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:31.448911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:31.448913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:31.449378 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:31.449561 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:31.449939 (+ 378us) server_negotiation.cc:299] Negotiation successful
0810 01:51:31.450235 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":171,"threads_started":1}
I20260810 01:51:31.454969 9526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 01:51:31.455083 9512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 920c769f59d045368940d5adad6cc4a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "920c769f59d045368940d5adad6cc4a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46567 } }
I20260810 01:51:31.455395 9512 raft_consensus.cc:374] T 00000000000000000000000000000000 P 920c769f59d045368940d5adad6cc4a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:31.455475 9512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 920c769f59d045368940d5adad6cc4a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 920c769f59d045368940d5adad6cc4a0, State: Initialized, Role: FOLLOWER
I20260810 01:51:31.456236 9512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 920c769f59d045368940d5adad6cc4a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "920c769f59d045368940d5adad6cc4a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46567 } }
I20260810 01:51:31.456457 9512 raft_consensus.cc:388] T 00000000000000000000000000000000 P 920c769f59d045368940d5adad6cc4a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:51:31.456544 9512 raft_consensus.cc:482] T 00000000000000000000000000000000 P 920c769f59d045368940d5adad6cc4a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:51:31.456686 9512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 920c769f59d045368940d5adad6cc4a0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:31.457924 9512 raft_consensus.cc:504] T 00000000000000000000000000000000 P 920c769f59d045368940d5adad6cc4a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "920c769f59d045368940d5adad6cc4a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46567 } }
I20260810 01:51:31.458516 9512 leader_election.cc:302] T 00000000000000000000000000000000 P 920c769f59d045368940d5adad6cc4a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 920c769f59d045368940d5adad6cc4a0; no voters:
I20260810 01:51:31.458922 9512 leader_election.cc:288] T 00000000000000000000000000000000 P 920c769f59d045368940d5adad6cc4a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:51:31.459053 9534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 920c769f59d045368940d5adad6cc4a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:31.459349 9534 raft_consensus.cc:686] T 00000000000000000000000000000000 P 920c769f59d045368940d5adad6cc4a0 [term 1 LEADER]: Becoming Leader. State: Replica: 920c769f59d045368940d5adad6cc4a0, State: Running, Role: LEADER
I20260810 01:51:31.459805 9534 consensus_queue.cc:237] T 00000000000000000000000000000000 P 920c769f59d045368940d5adad6cc4a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "920c769f59d045368940d5adad6cc4a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46567 } }
I20260810 01:51:31.460664 9512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 920c769f59d045368940d5adad6cc4a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:31.462894 9538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 920c769f59d045368940d5adad6cc4a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "920c769f59d045368940d5adad6cc4a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "920c769f59d045368940d5adad6cc4a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46567 } } }
I20260810 01:51:31.463030 9538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 920c769f59d045368940d5adad6cc4a0 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:31.463564 9536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 920c769f59d045368940d5adad6cc4a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 920c769f59d045368940d5adad6cc4a0. Latest consensus state: current_term: 1 leader_uuid: "920c769f59d045368940d5adad6cc4a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "920c769f59d045368940d5adad6cc4a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46567 } } }
I20260810 01:51:31.463917 9536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 920c769f59d045368940d5adad6cc4a0 [sys.catalog]: This master's current role is: LEADER
W20260810 01:51:31.467347 9553 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 920c769f59d045368940d5adad6cc4a0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 01:51:31.467437 9553 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 01:51:31.467571 9556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:51:31.468652 9556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:51:31.468662 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:31.457932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:46725 (local address 127.0.0.1:46567)
0810 01:51:31.458077 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:31.458082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:31.458333 (+ 251us) server_negotiation.cc:407] Connection header received
0810 01:51:31.458448 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:31.458451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:31.458507 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:31.458602 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:31.459622 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:31.460485 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:31.461627 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:31.462009 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:31.462128 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:31.464707 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:31.464711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:31.464714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:31.464748 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:31.466407 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:31.467025 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:31.467029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:31.467031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:31.467121 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:31.467541 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:31.467545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:31.467547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:31.467814 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:31.467979 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:31.468371 (+ 392us) server_negotiation.cc:299] Negotiation successful
0810 01:51:31.468490 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 01:51:31.474951 9556 catalog_manager.cc:1347] Generated new cluster ID: bac604d0490740b9a2933067af4932bb
I20260810 01:51:31.475039 9556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:51:31.489598 9556 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:51:31.490844 9556 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:51:31.496897 9556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 920c769f59d045368940d5adad6cc4a0: Generated new TSK 0
I20260810 01:51:31.497764 9556 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:51:31.508755 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 9400
W20260810 01:51:31.526232 9372 connection.cc:535] client connection to 127.0.0.1:46567 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46567): Connection reset by peer (error 104)
2026-08-10T01:51:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1818 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:51:31 dist-test-slave-kvfs krb5kdc[9561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:51:31 dist-test-slave-kvfs krb5kdc[9561](info): set up 2 sockets
Aug 10 01:51:31 dist-test-slave-kvfs krb5kdc[9561](info): commencing operation
krb5kdc: starting...
W20260810 01:51:32.802209 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.260s user 0.003s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:51:32 dist-test-slave-kvfs krb5kdc[9561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:51:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:51:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 01:51:32.955847 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32949
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:33.093008 9581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:33.093437 9581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:33.093535 9581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:33.098083 9581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:33.098202 9581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:33.098242 9581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:33.098307 9581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:33.098361 9581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:33.104497 9581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:33.106007 9581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:33.107380 9581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:33.118157 9589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:33.118157 9593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:33.118157 9590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:33.118655 9581 server_base.cc:1034] running on GCE node
I20260810 01:51:33.119414 9581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:33.120993 9581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:33.122182 9581 hybrid_clock.cc:648] HybridClock initialized: now 1786326693122158 us; error 55 us; skew 500 ppm
Aug 10 01:51:33 dist-test-slave-kvfs krb5kdc[9561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:33.131944 9581 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 01:51:33.133255 9581 webserver.cc:460] Webserver started at http://127.0.0.1:44803/ using document root <none> and password file <none>
I20260810 01:51:33.133975 9581 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:33.134042 9581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:33.134261 9581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:33.136999 9581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "c60d9ec865574c05bcb678ef779cde18"
format_stamp: "Formatted at 2026-08-10 01:51:33 on dist-test-slave-kvfs"
server_key: "ca6ddb3dcff097a7c25d684cdfa9b8da"
server_key_iv: "7a16770c249c9e6df2e68a5292cc06af"
server_key_version: "encryptionkey@0"
I20260810 01:51:33.137681 9581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "c60d9ec865574c05bcb678ef779cde18"
format_stamp: "Formatted at 2026-08-10 01:51:33 on dist-test-slave-kvfs"
server_key: "ca6ddb3dcff097a7c25d684cdfa9b8da"
server_key_iv: "7a16770c249c9e6df2e68a5292cc06af"
server_key_version: "encryptionkey@0"
I20260810 01:51:33.142462 9581 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 01:51:33.145705 9605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:33.147418 9581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 01:51:33.147624 9581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "c60d9ec865574c05bcb678ef779cde18"
format_stamp: "Formatted at 2026-08-10 01:51:33 on dist-test-slave-kvfs"
server_key: "ca6ddb3dcff097a7c25d684cdfa9b8da"
server_key_iv: "7a16770c249c9e6df2e68a5292cc06af"
server_key_version: "encryptionkey@0"
I20260810 01:51:33.147782 9581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:33.162788 9581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:33.166940 9581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:33.167201 9581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:33.176272 9581 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32949
I20260810 01:51:33.176316 9683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32949 every 8 connection(s)
I20260810 01:51:33.177551 9581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:51:33.181347 9684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:33.186241 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 9581
I20260810 01:51:33.186414 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:51:33.186735 9526 external_mini_cluster.cc:1428] Setting key e047f117e5dabd8de8774266f58392f0
I20260810 01:51:33.189177 9684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c60d9ec865574c05bcb678ef779cde18: Bootstrap starting.
I20260810 01:51:33.192638 9684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c60d9ec865574c05bcb678ef779cde18: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:33.193710 9684 log.cc:824] T 00000000000000000000000000000000 P c60d9ec865574c05bcb678ef779cde18: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:33.196669 9684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c60d9ec865574c05bcb678ef779cde18: No bootstrap required, opened a new log
I20260810 01:51:33.200196 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:33.188768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:49449 (local address 127.0.0.1:32949)
0810 01:51:33.189396 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:33.189419 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:33.189476 (+ 57us) server_negotiation.cc:407] Connection header received
0810 01:51:33.190423 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:33.190451 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:33.191029 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:33.191463 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:33.191970 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:33.193660 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:33.195056 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:33.195808 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:33.195988 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:33.197148 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:33.197158 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:33.197179 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:33.197830 (+ 651us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:33.197923 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:33.197961 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:33.198107 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:33.198332 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 01:51:33.198520 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":248,"threads_started":1}
I20260810 01:51:33.201406 9684 raft_consensus.cc:348] T 00000000000000000000000000000000 P c60d9ec865574c05bcb678ef779cde18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60d9ec865574c05bcb678ef779cde18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32949 } }
I20260810 01:51:33.201691 9684 raft_consensus.cc:374] T 00000000000000000000000000000000 P c60d9ec865574c05bcb678ef779cde18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:33.201773 9684 raft_consensus.cc:729] T 00000000000000000000000000000000 P c60d9ec865574c05bcb678ef779cde18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60d9ec865574c05bcb678ef779cde18, State: Initialized, Role: FOLLOWER
I20260810 01:51:33.202327 9684 consensus_queue.cc:260] T 00000000000000000000000000000000 P c60d9ec865574c05bcb678ef779cde18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60d9ec865574c05bcb678ef779cde18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32949 } }
I20260810 01:51:33.202517 9684 raft_consensus.cc:388] T 00000000000000000000000000000000 P c60d9ec865574c05bcb678ef779cde18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:51:33.202615 9684 raft_consensus.cc:482] T 00000000000000000000000000000000 P c60d9ec865574c05bcb678ef779cde18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:51:33.202747 9684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c60d9ec865574c05bcb678ef779cde18 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:33.204151 9526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 01:51:33.204118 9684 raft_consensus.cc:504] T 00000000000000000000000000000000 P c60d9ec865574c05bcb678ef779cde18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60d9ec865574c05bcb678ef779cde18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32949 } }
I20260810 01:51:33.204597 9684 leader_election.cc:302] T 00000000000000000000000000000000 P c60d9ec865574c05bcb678ef779cde18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c60d9ec865574c05bcb678ef779cde18; no voters:
I20260810 01:51:33.205044 9684 leader_election.cc:288] T 00000000000000000000000000000000 P c60d9ec865574c05bcb678ef779cde18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:51:33.205184 9693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c60d9ec865574c05bcb678ef779cde18 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:33.205461 9693 raft_consensus.cc:686] T 00000000000000000000000000000000 P c60d9ec865574c05bcb678ef779cde18 [term 1 LEADER]: Becoming Leader. State: Replica: c60d9ec865574c05bcb678ef779cde18, State: Running, Role: LEADER
I20260810 01:51:33.205933 9693 consensus_queue.cc:237] T 00000000000000000000000000000000 P c60d9ec865574c05bcb678ef779cde18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60d9ec865574c05bcb678ef779cde18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32949 } }
I20260810 01:51:33.206877 9684 sys_catalog.cc:564] T 00000000000000000000000000000000 P c60d9ec865574c05bcb678ef779cde18 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:33.208243 9696 sys_catalog.cc:455] T 00000000000000000000000000000000 P c60d9ec865574c05bcb678ef779cde18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c60d9ec865574c05bcb678ef779cde18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60d9ec865574c05bcb678ef779cde18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32949 } } }
I20260810 01:51:33.208392 9696 sys_catalog.cc:458] T 00000000000000000000000000000000 P c60d9ec865574c05bcb678ef779cde18 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:33.208702 9697 sys_catalog.cc:455] T 00000000000000000000000000000000 P c60d9ec865574c05bcb678ef779cde18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c60d9ec865574c05bcb678ef779cde18. Latest consensus state: current_term: 1 leader_uuid: "c60d9ec865574c05bcb678ef779cde18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60d9ec865574c05bcb678ef779cde18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32949 } } }
I20260810 01:51:33.208786 9697 sys_catalog.cc:458] T 00000000000000000000000000000000 P c60d9ec865574c05bcb678ef779cde18 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:33.208901 9712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:51:33.213207 9712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:51:33.215471 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:33.207108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:41505 (local address 127.0.0.1:32949)
0810 01:51:33.207261 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:33.207268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:33.209832 (+ 2564us) server_negotiation.cc:407] Connection header received
0810 01:51:33.210095 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:33.210098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:33.210163 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:33.210255 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:33.210658 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:33.211468 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:33.212782 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:33.213119 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:33.213203 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:33.214819 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:33.214823 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:33.214826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:33.214843 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:33.214878 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:33.214893 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:33.214960 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:33.215242 (+ 282us) server_negotiation.cc:299] Negotiation successful
0810 01:51:33.215290 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 01:51:33.220625 9712 catalog_manager.cc:1347] Generated new cluster ID: a690e65fe5764797adf5f86506036b99
I20260810 01:51:33.220717 9712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:51:33.231319 9712 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:51:33.232841 9712 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:51:33.242623 9712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c60d9ec865574c05bcb678ef779cde18: Generated new TSK 0
I20260810 01:51:33.243891 9712 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:51:33.252938 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 9581
W20260810 01:51:33.275642 9559 connection.cc:535] client connection to 127.0.0.1:32949 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32949): Connection reset by peer (error 104)
2026-08-10T01:51:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1749 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:51:33 dist-test-slave-kvfs krb5kdc[9730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:51:33 dist-test-slave-kvfs krb5kdc[9730](info): set up 2 sockets
Aug 10 01:51:33 dist-test-slave-kvfs krb5kdc[9730](info): commencing operation
krb5kdc: starting...
W20260810 01:51:34.553622 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.262s user 0.005s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:51:34 dist-test-slave-kvfs krb5kdc[9730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:51:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:51:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 01:51:34.700629 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43709
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:34.835934 9749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:34.836380 9749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:34.836432 9749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:34.840958 9749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:34.841053 9749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:34.841073 9749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:34.841091 9749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:34.841107 9749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:34.847211 9749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:34.848906 9749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:34.850332 9749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:34.860931 9756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:34.861096 9757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:34.861321 9749 server_base.cc:1034] running on GCE node
W20260810 01:51:34.860932 9760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:34.862216 9749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:34.863685 9749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:34.864866 9749 hybrid_clock.cc:648] HybridClock initialized: now 1786326694864829 us; error 55 us; skew 500 ppm
Aug 10 01:51:34 dist-test-slave-kvfs krb5kdc[9730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:34.868590 9749 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 01:51:34.869966 9749 webserver.cc:460] Webserver started at http://127.0.0.1:35197/ using document root <none> and password file <none>
I20260810 01:51:34.870752 9749 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:34.870831 9749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:34.871129 9749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:34.874086 9749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "f21be97bba6c403da01fec60a5552bdd"
format_stamp: "Formatted at 2026-08-10 01:51:34 on dist-test-slave-kvfs"
server_key: "4ac020d09a7352b8bcea6a826e49500d"
server_key_iv: "19c2011889fa542428b7313ca9931011"
server_key_version: "encryptionkey@0"
I20260810 01:51:34.874831 9749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "f21be97bba6c403da01fec60a5552bdd"
format_stamp: "Formatted at 2026-08-10 01:51:34 on dist-test-slave-kvfs"
server_key: "4ac020d09a7352b8bcea6a826e49500d"
server_key_iv: "19c2011889fa542428b7313ca9931011"
server_key_version: "encryptionkey@0"
I20260810 01:51:34.879806 9749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260810 01:51:34.882999 9770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:34.884764 9749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:51:34.884929 9749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "f21be97bba6c403da01fec60a5552bdd"
format_stamp: "Formatted at 2026-08-10 01:51:34 on dist-test-slave-kvfs"
server_key: "4ac020d09a7352b8bcea6a826e49500d"
server_key_iv: "19c2011889fa542428b7313ca9931011"
server_key_version: "encryptionkey@0"
I20260810 01:51:34.885125 9749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:34.890893 9749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:34.894881 9749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:34.895113 9749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:34.904300 9749 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43709
I20260810 01:51:34.904386 9842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43709 every 8 connection(s)
I20260810 01:51:34.905565 9749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:51:34.909368 9843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:34.910548 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 9749
I20260810 01:51:34.910705 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:51:34.911101 9526 external_mini_cluster.cc:1428] Setting key 60ea0afab059789296c040a844637a27
I20260810 01:51:34.917516 9846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:34.912839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:48943 (local address 127.0.0.1:43709)
0810 01:51:34.913348 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:34.913374 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:34.913420 (+ 46us) server_negotiation.cc:407] Connection header received
0810 01:51:34.914171 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:34.914191 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:34.914546 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:34.914887 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:34.915897 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:34.915903 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:34.915917 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:34.915929 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:34.916001 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:34.916075 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:34.916222 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:34.916353 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 01:51:34.916570 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":171,"threads_started":1}
I20260810 01:51:34.917630 9843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f21be97bba6c403da01fec60a5552bdd: Bootstrap starting.
I20260810 01:51:34.920883 9843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f21be97bba6c403da01fec60a5552bdd: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:34.921115 9526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 01:51:34.922034 9843 log.cc:824] T 00000000000000000000000000000000 P f21be97bba6c403da01fec60a5552bdd: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:34.924975 9843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f21be97bba6c403da01fec60a5552bdd: No bootstrap required, opened a new log
I20260810 01:51:34.926292 9846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:34.924096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:53349 (local address 127.0.0.1:43709)
0810 01:51:34.924233 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:34.924238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:34.924504 (+ 266us) server_negotiation.cc:407] Connection header received
0810 01:51:34.924676 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:34.924679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:34.924725 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:34.924854 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:34.925808 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:34.925812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:34.925815 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:34.925835 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:34.925866 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:34.925878 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:34.926013 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:34.926096 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 01:51:34.926150 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 01:51:34.930394 9843 raft_consensus.cc:348] T 00000000000000000000000000000000 P f21be97bba6c403da01fec60a5552bdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21be97bba6c403da01fec60a5552bdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43709 } }
I20260810 01:51:34.930724 9843 raft_consensus.cc:374] T 00000000000000000000000000000000 P f21be97bba6c403da01fec60a5552bdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:34.930788 9843 raft_consensus.cc:729] T 00000000000000000000000000000000 P f21be97bba6c403da01fec60a5552bdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21be97bba6c403da01fec60a5552bdd, State: Initialized, Role: FOLLOWER
I20260810 01:51:34.931376 9843 consensus_queue.cc:260] T 00000000000000000000000000000000 P f21be97bba6c403da01fec60a5552bdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21be97bba6c403da01fec60a5552bdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43709 } }
I20260810 01:51:34.931536 9843 raft_consensus.cc:388] T 00000000000000000000000000000000 P f21be97bba6c403da01fec60a5552bdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:51:34.931629 9843 raft_consensus.cc:482] T 00000000000000000000000000000000 P f21be97bba6c403da01fec60a5552bdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:51:34.931766 9843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f21be97bba6c403da01fec60a5552bdd [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:34.933061 9843 raft_consensus.cc:504] T 00000000000000000000000000000000 P f21be97bba6c403da01fec60a5552bdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21be97bba6c403da01fec60a5552bdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43709 } }
I20260810 01:51:34.933511 9843 leader_election.cc:302] T 00000000000000000000000000000000 P f21be97bba6c403da01fec60a5552bdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f21be97bba6c403da01fec60a5552bdd; no voters:
I20260810 01:51:34.933910 9843 leader_election.cc:288] T 00000000000000000000000000000000 P f21be97bba6c403da01fec60a5552bdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:51:34.934059 9864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f21be97bba6c403da01fec60a5552bdd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:34.934417 9864 raft_consensus.cc:686] T 00000000000000000000000000000000 P f21be97bba6c403da01fec60a5552bdd [term 1 LEADER]: Becoming Leader. State: Replica: f21be97bba6c403da01fec60a5552bdd, State: Running, Role: LEADER
I20260810 01:51:34.934762 9864 consensus_queue.cc:237] T 00000000000000000000000000000000 P f21be97bba6c403da01fec60a5552bdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21be97bba6c403da01fec60a5552bdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43709 } }
I20260810 01:51:34.935750 9843 sys_catalog.cc:564] T 00000000000000000000000000000000 P f21be97bba6c403da01fec60a5552bdd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:34.937367 9865 sys_catalog.cc:455] T 00000000000000000000000000000000 P f21be97bba6c403da01fec60a5552bdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f21be97bba6c403da01fec60a5552bdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21be97bba6c403da01fec60a5552bdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43709 } } }
I20260810 01:51:34.937511 9865 sys_catalog.cc:458] T 00000000000000000000000000000000 P f21be97bba6c403da01fec60a5552bdd [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:34.937911 9866 sys_catalog.cc:455] T 00000000000000000000000000000000 P f21be97bba6c403da01fec60a5552bdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f21be97bba6c403da01fec60a5552bdd. Latest consensus state: current_term: 1 leader_uuid: "f21be97bba6c403da01fec60a5552bdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f21be97bba6c403da01fec60a5552bdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43709 } } }
I20260810 01:51:34.938014 9866 sys_catalog.cc:458] T 00000000000000000000000000000000 P f21be97bba6c403da01fec60a5552bdd [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:34.937989 9875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:51:34.941773 9875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:51:34.948545 9875 catalog_manager.cc:1347] Generated new cluster ID: 6f6be32178ec41688bbeddfe649a0680
I20260810 01:51:34.948642 9875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:51:34.967711 9875 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:51:34.968197 9875 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:51:34.992759 9875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f21be97bba6c403da01fec60a5552bdd: Generated new TSK 0
I20260810 01:51:34.993806 9875 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:51:35.004524 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 9749
2026-08-10T01:51:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1747 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:51:35 dist-test-slave-kvfs krb5kdc[9896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:51:35 dist-test-slave-kvfs krb5kdc[9896](info): set up 2 sockets
Aug 10 01:51:35 dist-test-slave-kvfs krb5kdc[9896](info): commencing operation
krb5kdc: starting...
W20260810 01:51:36.353237 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.313s user 0.009s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:51:36 dist-test-slave-kvfs krb5kdc[9896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:51:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:51:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 01:51:36.512458 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36357
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:36.652046 9917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:36.652525 9917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:36.652621 9917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:36.658143 9917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:36.658303 9917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:36.658352 9917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:36.658399 9917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:36.658434 9917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:36.665014 9917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36357
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:36.666654 9917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:36.668095 9917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:36.679171 9925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:36.679178 9923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:36.679171 9922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:36.679654 9917 server_base.cc:1034] running on GCE node
I20260810 01:51:36.680517 9917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:36.682096 9917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:36.683288 9917 hybrid_clock.cc:648] HybridClock initialized: now 1786326696683254 us; error 46 us; skew 500 ppm
Aug 10 01:51:36 dist-test-slave-kvfs krb5kdc[9896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:36.687001 9917 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 01:51:36.688282 9917 webserver.cc:460] Webserver started at http://127.0.0.1:41513/ using document root <none> and password file <none>
I20260810 01:51:36.688993 9917 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:36.689062 9917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:36.689284 9917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:36.691955 9917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "07fffbc7784142f2894004aabe561e59"
format_stamp: "Formatted at 2026-08-10 01:51:36 on dist-test-slave-kvfs"
server_key: "b9eba9bcb11d1a45908dd4016bee1bf9"
server_key_iv: "5699a47e62ee9717bd38de4085ab0769"
server_key_version: "encryptionkey@0"
I20260810 01:51:36.692711 9917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "07fffbc7784142f2894004aabe561e59"
format_stamp: "Formatted at 2026-08-10 01:51:36 on dist-test-slave-kvfs"
server_key: "b9eba9bcb11d1a45908dd4016bee1bf9"
server_key_iv: "5699a47e62ee9717bd38de4085ab0769"
server_key_version: "encryptionkey@0"
I20260810 01:51:36.697440 9917 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 01:51:36.700635 9936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:36.702215 9917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:51:36.702391 9917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "07fffbc7784142f2894004aabe561e59"
format_stamp: "Formatted at 2026-08-10 01:51:36 on dist-test-slave-kvfs"
server_key: "b9eba9bcb11d1a45908dd4016bee1bf9"
server_key_iv: "5699a47e62ee9717bd38de4085ab0769"
server_key_version: "encryptionkey@0"
I20260810 01:51:36.702543 9917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:36.730211 9917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:36.734265 9917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:36.734532 9917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:36.743824 9917 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36357
I20260810 01:51:36.743855 10022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36357 every 8 connection(s)
I20260810 01:51:36.745175 9917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:51:36.748916 10023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:36.753185 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 9917
I20260810 01:51:36.753360 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:51:36.753743 9526 external_mini_cluster.cc:1428] Setting key 93c183969b37306fbaa7fe2b41c431d3
I20260810 01:51:36.756995 10023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07fffbc7784142f2894004aabe561e59: Bootstrap starting.
I20260810 01:51:36.759913 10023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07fffbc7784142f2894004aabe561e59: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:36.760895 10023 log.cc:824] T 00000000000000000000000000000000 P 07fffbc7784142f2894004aabe561e59: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:36.763252 10023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07fffbc7784142f2894004aabe561e59: No bootstrap required, opened a new log
Aug 10 01:51:36 dist-test-slave-kvfs krb5kdc[9896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 01:51:36.767280 10023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07fffbc7784142f2894004aabe561e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07fffbc7784142f2894004aabe561e59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36357 } }
I20260810 01:51:36.767627 10023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07fffbc7784142f2894004aabe561e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:36.767716 10023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07fffbc7784142f2894004aabe561e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07fffbc7784142f2894004aabe561e59, State: Initialized, Role: FOLLOWER
I20260810 01:51:36.768391 10023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07fffbc7784142f2894004aabe561e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07fffbc7784142f2894004aabe561e59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36357 } }
I20260810 01:51:36.768607 10023 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07fffbc7784142f2894004aabe561e59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:51:36.768705 10023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07fffbc7784142f2894004aabe561e59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:51:36.768847 10023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07fffbc7784142f2894004aabe561e59 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:36.770117 10023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07fffbc7784142f2894004aabe561e59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07fffbc7784142f2894004aabe561e59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36357 } }
I20260810 01:51:36.770603 10023 leader_election.cc:302] T 00000000000000000000000000000000 P 07fffbc7784142f2894004aabe561e59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07fffbc7784142f2894004aabe561e59; no voters:
I20260810 01:51:36.771050 10023 leader_election.cc:288] T 00000000000000000000000000000000 P 07fffbc7784142f2894004aabe561e59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:51:36.771183 10033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07fffbc7784142f2894004aabe561e59 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:36.771418 10033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07fffbc7784142f2894004aabe561e59 [term 1 LEADER]: Becoming Leader. State: Replica: 07fffbc7784142f2894004aabe561e59, State: Running, Role: LEADER
I20260810 01:51:36.771788 10033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07fffbc7784142f2894004aabe561e59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07fffbc7784142f2894004aabe561e59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36357 } }
I20260810 01:51:36.772156 10030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:36.755687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:56623 (local address 127.0.0.1:36357)
0810 01:51:36.756282 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:36.756301 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:36.756346 (+ 45us) server_negotiation.cc:407] Connection header received
0810 01:51:36.757117 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:36.757170 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:36.757635 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:36.758046 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:36.759011 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:36.760242 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:36.761240 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:36.762008 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:36.762201 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:36.765200 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:36.765206 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:36.765223 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:36.765254 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:36.767942 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:36.768654 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:36.768663 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:36.768673 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:36.768867 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:36.769312 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:36.769314 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:36.769316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:36.769814 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:36.770014 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:36.770506 (+ 492us) server_negotiation.cc:299] Negotiation successful
0810 01:51:36.770849 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":207,"threads_started":1}
I20260810 01:51:36.772791 10023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07fffbc7784142f2894004aabe561e59 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:36.773794 10035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07fffbc7784142f2894004aabe561e59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07fffbc7784142f2894004aabe561e59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07fffbc7784142f2894004aabe561e59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36357 } } }
I20260810 01:51:36.773950 10035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07fffbc7784142f2894004aabe561e59 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:36.774361 10036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07fffbc7784142f2894004aabe561e59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07fffbc7784142f2894004aabe561e59. Latest consensus state: current_term: 1 leader_uuid: "07fffbc7784142f2894004aabe561e59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07fffbc7784142f2894004aabe561e59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36357 } } }
I20260810 01:51:36.774466 10036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07fffbc7784142f2894004aabe561e59 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:36.774430 10040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:51:36.776057 9526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 01:51:36.778610 10040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:51:36.788739 10040 catalog_manager.cc:1347] Generated new cluster ID: 485d87d22ca24973a6e146a30084b9f5
I20260810 01:51:36.788794 10030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:36.778890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:41345 (local address 127.0.0.1:36357)
0810 01:51:36.779023 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:36.779028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:36.779823 (+ 795us) server_negotiation.cc:407] Connection header received
0810 01:51:36.780061 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:36.780067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:36.780130 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:36.780223 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:36.781005 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:36.781581 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:36.782457 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:36.782759 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:36.782876 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:36.784838 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:36.784842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:36.784845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:36.784874 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:36.786639 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:36.787167 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:36.787171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:36.787173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:36.787276 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:36.787649 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:36.787653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:36.787655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:36.787877 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:36.788122 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:36.788479 (+ 357us) server_negotiation.cc:299] Negotiation successful
0810 01:51:36.788609 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 01:51:36.788828 10040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:51:36.801812 10040 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:51:36.803217 10040 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:51:36.819829 10040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07fffbc7784142f2894004aabe561e59: Generated new TSK 0
I20260810 01:51:36.821192 10040 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:51:36.836126 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 9917
W20260810 01:51:36.850904 9894 connection.cc:535] client connection to 127.0.0.1:36357 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36357): Connection reset by peer (error 104)
2026-08-10T01:51:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1826 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:51:36 dist-test-slave-kvfs krb5kdc[10073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:51:36 dist-test-slave-kvfs krb5kdc[10073](info): set up 2 sockets
Aug 10 01:51:36 dist-test-slave-kvfs krb5kdc[10073](info): commencing operation
krb5kdc: starting...
W20260810 01:51:38.149883 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.283s user 0.002s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:51:38 dist-test-slave-kvfs krb5kdc[10073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:51:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:51:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 01:51:38.308354 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37375
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:38.450439 10095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:38.450876 10095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:38.450965 10095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:38.455350 10095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:38.455487 10095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:38.455525 10095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:38.455564 10095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:38.455605 10095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:38.461714 10095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37375
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:38.463266 10095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:38.464699 10095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:38.476085 10102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:38.476092 10103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:38.476263 10095 server_base.cc:1034] running on GCE node
W20260810 01:51:38.476516 10107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:38.477285 10095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:38.478709 10095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:38.479887 10095 hybrid_clock.cc:648] HybridClock initialized: now 1786326698479860 us; error 49 us; skew 500 ppm
Aug 10 01:51:38 dist-test-slave-kvfs krb5kdc[10073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:38.487458 10095 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 01:51:38.488835 10095 webserver.cc:460] Webserver started at http://127.0.0.1:43829/ using document root <none> and password file <none>
I20260810 01:51:38.489607 10095 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:38.489711 10095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:38.490026 10095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:38.492779 10095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "6b0cd864e9ab4692a18631b33c0c10f4"
format_stamp: "Formatted at 2026-08-10 01:51:38 on dist-test-slave-kvfs"
server_key: "d939de9331485efdd946d44db0854898"
server_key_iv: "2e9ca2fd109121d460c71090ded8b8c8"
server_key_version: "encryptionkey@0"
I20260810 01:51:38.493490 10095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "6b0cd864e9ab4692a18631b33c0c10f4"
format_stamp: "Formatted at 2026-08-10 01:51:38 on dist-test-slave-kvfs"
server_key: "d939de9331485efdd946d44db0854898"
server_key_iv: "2e9ca2fd109121d460c71090ded8b8c8"
server_key_version: "encryptionkey@0"
I20260810 01:51:38.498471 10095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260810 01:51:38.501622 10115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:38.503176 10095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260810 01:51:38.503350 10095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "6b0cd864e9ab4692a18631b33c0c10f4"
format_stamp: "Formatted at 2026-08-10 01:51:38 on dist-test-slave-kvfs"
server_key: "d939de9331485efdd946d44db0854898"
server_key_iv: "2e9ca2fd109121d460c71090ded8b8c8"
server_key_version: "encryptionkey@0"
I20260810 01:51:38.503500 10095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:38.516923 10095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:38.521230 10095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:38.521502 10095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:38.531126 10095 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37375
I20260810 01:51:38.531154 10197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37375 every 8 connection(s)
I20260810 01:51:38.532516 10095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:51:38.536283 10199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:38.539163 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 10095
I20260810 01:51:38.539341 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:51:38.539675 9526 external_mini_cluster.cc:1428] Setting key f313f4b91b6274d7f36cfe679aaf62b2
I20260810 01:51:38.544529 10199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b0cd864e9ab4692a18631b33c0c10f4: Bootstrap starting.
I20260810 01:51:38.547578 10199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b0cd864e9ab4692a18631b33c0c10f4: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:38.548635 10199 log.cc:824] T 00000000000000000000000000000000 P 6b0cd864e9ab4692a18631b33c0c10f4: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:38.551028 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:38.541734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:39623 (local address 127.0.0.1:37375)
0810 01:51:38.542321 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:38.542341 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:38.542389 (+ 48us) server_negotiation.cc:407] Connection header received
0810 01:51:38.543222 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:38.543256 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:38.543712 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:38.544168 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:38.544672 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:38.545841 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:38.546830 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:38.547397 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:38.547623 (+ 226us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:38.548712 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:38.548718 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:38.548736 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:38.549364 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:38.549445 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:38.549484 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:38.549646 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:38.549899 (+ 253us) server_negotiation.cc:299] Negotiation successful
0810 01:51:38.550084 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":172,"threads_started":1}
I20260810 01:51:38.551348 10199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b0cd864e9ab4692a18631b33c0c10f4: No bootstrap required, opened a new log
I20260810 01:51:38.554943 9526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 01:51:38.556845 10199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b0cd864e9ab4692a18631b33c0c10f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0cd864e9ab4692a18631b33c0c10f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37375 } }
I20260810 01:51:38.557210 10199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b0cd864e9ab4692a18631b33c0c10f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:38.557307 10199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b0cd864e9ab4692a18631b33c0c10f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b0cd864e9ab4692a18631b33c0c10f4, State: Initialized, Role: FOLLOWER
I20260810 01:51:38.557998 10199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b0cd864e9ab4692a18631b33c0c10f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0cd864e9ab4692a18631b33c0c10f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37375 } }
I20260810 01:51:38.558235 10199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b0cd864e9ab4692a18631b33c0c10f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:51:38.558394 10199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b0cd864e9ab4692a18631b33c0c10f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:51:38.558545 10199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b0cd864e9ab4692a18631b33c0c10f4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:38.560091 10199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b0cd864e9ab4692a18631b33c0c10f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0cd864e9ab4692a18631b33c0c10f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37375 } }
I20260810 01:51:38.560587 10199 leader_election.cc:302] T 00000000000000000000000000000000 P 6b0cd864e9ab4692a18631b33c0c10f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b0cd864e9ab4692a18631b33c0c10f4; no voters:
I20260810 01:51:38.561103 10199 leader_election.cc:288] T 00000000000000000000000000000000 P 6b0cd864e9ab4692a18631b33c0c10f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:51:38.561196 10215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b0cd864e9ab4692a18631b33c0c10f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:38.561452 10215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b0cd864e9ab4692a18631b33c0c10f4 [term 1 LEADER]: Becoming Leader. State: Replica: 6b0cd864e9ab4692a18631b33c0c10f4, State: Running, Role: LEADER
I20260810 01:51:38.561921 10215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b0cd864e9ab4692a18631b33c0c10f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0cd864e9ab4692a18631b33c0c10f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37375 } }
I20260810 01:51:38.562955 10199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b0cd864e9ab4692a18631b33c0c10f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:38.563141 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:38.557766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:38275 (local address 127.0.0.1:37375)
0810 01:51:38.557930 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:38.557935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:38.558157 (+ 222us) server_negotiation.cc:407] Connection header received
0810 01:51:38.558331 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:38.558336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:38.558399 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:38.558513 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:38.558923 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:38.559506 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:38.560875 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:38.561146 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:38.562479 (+ 1333us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:38.562623 (+ 144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:38.562626 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:38.562628 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:38.562639 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:38.562657 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:38.562667 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:38.562775 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:38.562932 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 01:51:38.562975 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 01:51:38.563531 10215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b0cd864e9ab4692a18631b33c0c10f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b0cd864e9ab4692a18631b33c0c10f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0cd864e9ab4692a18631b33c0c10f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37375 } } }
I20260810 01:51:38.563663 10215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b0cd864e9ab4692a18631b33c0c10f4 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:38.564081 10216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b0cd864e9ab4692a18631b33c0c10f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b0cd864e9ab4692a18631b33c0c10f4. Latest consensus state: current_term: 1 leader_uuid: "6b0cd864e9ab4692a18631b33c0c10f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b0cd864e9ab4692a18631b33c0c10f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37375 } } }
I20260810 01:51:38.564179 10216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b0cd864e9ab4692a18631b33c0c10f4 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:38.564778 10219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:51:38.569118 10219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:51:38.576344 10219 catalog_manager.cc:1347] Generated new cluster ID: a9152a0ccd084ec9a06ca76166b81855
I20260810 01:51:38.576458 10219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:51:38.595785 10219 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:51:38.597146 10219 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:51:38.604674 10219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b0cd864e9ab4692a18631b33c0c10f4: Generated new TSK 0
I20260810 01:51:38.605836 10219 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:51:38.620188 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 10095
W20260810 01:51:38.641683 10068 connection.cc:535] client connection to 127.0.0.1:37375 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37375): Connection reset by peer (error 104)
2026-08-10T01:51:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1791 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:51:38 dist-test-slave-kvfs krb5kdc[10243](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:51:38 dist-test-slave-kvfs krb5kdc[10243](info): set up 2 sockets
Aug 10 01:51:38 dist-test-slave-kvfs krb5kdc[10243](info): commencing operation
krb5kdc: starting...
W20260810 01:51:39.994454 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.336s user 0.001s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:51:40 dist-test-slave-kvfs krb5kdc[10243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:51:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:51:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 01:51:40.148438 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43449
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:40.286021 10261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:40.286414 10261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:40.286468 10261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:40.291221 10261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:40.291303 10261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:40.291324 10261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:40.291342 10261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:40.291358 10261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:40.297394 10261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43449
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786326392877349-9526-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:40.298841 10261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:40.300215 10261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:40.310868 10271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:40.310868 10273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:40.310885 10270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:40.310968 10261 server_base.cc:1034] running on GCE node
I20260810 01:51:40.311952 10261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:40.313442 10261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:40.314631 10261 hybrid_clock.cc:648] HybridClock initialized: now 1786326700314613 us; error 30 us; skew 500 ppm
Aug 10 01:51:40 dist-test-slave-kvfs krb5kdc[10243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:40.318061 10261 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 01:51:40.319197 10261 webserver.cc:460] Webserver started at http://127.0.0.1:43493/ using document root <none> and password file <none>
I20260810 01:51:40.319916 10261 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:40.320029 10261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:40.320328 10261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:40.323145 10261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "816b3da48a0a4c20ae857ed4774377ac"
format_stamp: "Formatted at 2026-08-10 01:51:40 on dist-test-slave-kvfs"
server_key: "0ff886251914328785d8ff458ebbd609"
server_key_iv: "9c5b6c70149f558faaff69b4e2d134f2"
server_key_version: "encryptionkey@0"
I20260810 01:51:40.323803 10261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "816b3da48a0a4c20ae857ed4774377ac"
format_stamp: "Formatted at 2026-08-10 01:51:40 on dist-test-slave-kvfs"
server_key: "0ff886251914328785d8ff458ebbd609"
server_key_iv: "9c5b6c70149f558faaff69b4e2d134f2"
server_key_version: "encryptionkey@0"
I20260810 01:51:40.328518 10261 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260810 01:51:40.331449 10284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:40.333127 10261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:51:40.333294 10261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "816b3da48a0a4c20ae857ed4774377ac"
format_stamp: "Formatted at 2026-08-10 01:51:40 on dist-test-slave-kvfs"
server_key: "0ff886251914328785d8ff458ebbd609"
server_key_iv: "9c5b6c70149f558faaff69b4e2d134f2"
server_key_version: "encryptionkey@0"
I20260810 01:51:40.333438 10261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:40.339190 10261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:40.342810 10261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:40.343065 10261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:40.351850 10261 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43449
I20260810 01:51:40.351869 10360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43449 every 8 connection(s)
I20260810 01:51:40.353114 10261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:51:40.356820 10361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:40.358278 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 10261
I20260810 01:51:40.358445 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:51:40.358839 9526 external_mini_cluster.cc:1428] Setting key 25d2ac0f333e18adaff2d56fa491fc23
I20260810 01:51:40.365110 10361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 816b3da48a0a4c20ae857ed4774377ac: Bootstrap starting.
I20260810 01:51:40.365312 10365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:40.360700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:36169 (local address 127.0.0.1:43449)
0810 01:51:40.361254 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:40.361273 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:40.361316 (+ 43us) server_negotiation.cc:407] Connection header received
0810 01:51:40.362061 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:40.362082 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:40.362442 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:40.362845 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:40.363793 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:40.363798 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:40.363812 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:40.363824 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:40.363889 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:40.363911 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:40.364112 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:40.364194 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 01:51:40.364389 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":155,"threads_started":1}
I20260810 01:51:40.368414 10361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 816b3da48a0a4c20ae857ed4774377ac: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:40.368669 9526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 01:51:40.369417 10361 log.cc:824] T 00000000000000000000000000000000 P 816b3da48a0a4c20ae857ed4774377ac: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:40.372253 10361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 816b3da48a0a4c20ae857ed4774377ac: No bootstrap required, opened a new log
I20260810 01:51:40.373202 10365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:40.371284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:55927 (local address 127.0.0.1:43449)
0810 01:51:40.371420 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:40.371425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:40.371638 (+ 213us) server_negotiation.cc:407] Connection header received
0810 01:51:40.371753 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:40.371756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:40.371803 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:40.371908 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:40.372786 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:40.372790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:40.372793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:40.372806 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:40.372846 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:40.372861 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:40.372953 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:40.373025 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 01:51:40.373070 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 01:51:40.376791 10361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 816b3da48a0a4c20ae857ed4774377ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816b3da48a0a4c20ae857ed4774377ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43449 } }
I20260810 01:51:40.377092 10361 raft_consensus.cc:374] T 00000000000000000000000000000000 P 816b3da48a0a4c20ae857ed4774377ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:40.377157 10361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 816b3da48a0a4c20ae857ed4774377ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816b3da48a0a4c20ae857ed4774377ac, State: Initialized, Role: FOLLOWER
I20260810 01:51:40.377749 10361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 816b3da48a0a4c20ae857ed4774377ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816b3da48a0a4c20ae857ed4774377ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43449 } }
I20260810 01:51:40.377934 10361 raft_consensus.cc:388] T 00000000000000000000000000000000 P 816b3da48a0a4c20ae857ed4774377ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:51:40.378018 10361 raft_consensus.cc:482] T 00000000000000000000000000000000 P 816b3da48a0a4c20ae857ed4774377ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:51:40.378141 10361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 816b3da48a0a4c20ae857ed4774377ac [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:40.379381 10361 raft_consensus.cc:504] T 00000000000000000000000000000000 P 816b3da48a0a4c20ae857ed4774377ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816b3da48a0a4c20ae857ed4774377ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43449 } }
I20260810 01:51:40.379822 10361 leader_election.cc:302] T 00000000000000000000000000000000 P 816b3da48a0a4c20ae857ed4774377ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 816b3da48a0a4c20ae857ed4774377ac; no voters:
I20260810 01:51:40.380272 10361 leader_election.cc:288] T 00000000000000000000000000000000 P 816b3da48a0a4c20ae857ed4774377ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:51:40.380427 10375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 816b3da48a0a4c20ae857ed4774377ac [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:40.380688 10375 raft_consensus.cc:686] T 00000000000000000000000000000000 P 816b3da48a0a4c20ae857ed4774377ac [term 1 LEADER]: Becoming Leader. State: Replica: 816b3da48a0a4c20ae857ed4774377ac, State: Running, Role: LEADER
I20260810 01:51:40.381141 10375 consensus_queue.cc:237] T 00000000000000000000000000000000 P 816b3da48a0a4c20ae857ed4774377ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816b3da48a0a4c20ae857ed4774377ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43449 } }
I20260810 01:51:40.382025 10361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 816b3da48a0a4c20ae857ed4774377ac [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:40.383313 10377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 816b3da48a0a4c20ae857ed4774377ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 816b3da48a0a4c20ae857ed4774377ac. Latest consensus state: current_term: 1 leader_uuid: "816b3da48a0a4c20ae857ed4774377ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816b3da48a0a4c20ae857ed4774377ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43449 } } }
I20260810 01:51:40.383476 10377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 816b3da48a0a4c20ae857ed4774377ac [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:40.383778 10376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 816b3da48a0a4c20ae857ed4774377ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "816b3da48a0a4c20ae857ed4774377ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816b3da48a0a4c20ae857ed4774377ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43449 } } }
I20260810 01:51:40.383915 10376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 816b3da48a0a4c20ae857ed4774377ac [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:40.383908 10387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:51:40.387564 10387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:51:40.394526 10387 catalog_manager.cc:1347] Generated new cluster ID: a007448ff8e24388a3c1abad5fdde237
I20260810 01:51:40.394623 10387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:51:40.415711 10387 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:51:40.416224 10387 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:51:40.423233 10387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 816b3da48a0a4c20ae857ed4774377ac: Generated new TSK 0
I20260810 01:51:40.424240 10387 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:51:40.427726 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 10261
2026-08-10T01:51:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1806 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:51:40 dist-test-slave-kvfs krb5kdc[10411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:51:40 dist-test-slave-kvfs krb5kdc[10411](info): set up 2 sockets
Aug 10 01:51:40 dist-test-slave-kvfs krb5kdc[10411](info): commencing operation
krb5kdc: starting...
W20260810 01:51:41.724293 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.259s user 0.001s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:51:41 dist-test-slave-kvfs krb5kdc[10411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:51:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:51:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:51:41.875536 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:45167
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:35107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:45167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:42.014107 10431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:42.014614 10431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:42.014711 10431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:42.019241 10431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:42.019320 10431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:42.019340 10431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:42.019357 10431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:42.019373 10431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:42.025207 10431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:45167
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:45167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:42.026547 10431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:42.027829 10431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:42.037990 10448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:42.037990 10443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:42.038020 10442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:42.038470 10431 server_base.cc:1034] running on GCE node
I20260810 01:51:42.039227 10431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:42.040680 10431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:42.041877 10431 hybrid_clock.cc:648] HybridClock initialized: now 1786326702041861 us; error 57 us; skew 500 ppm
Aug 10 01:51:42 dist-test-slave-kvfs krb5kdc[10411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326702, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:42.045284 10431 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:51:42.046414 10431 webserver.cc:460] Webserver started at http://127.9.77.190:42441/ using document root <none> and password file <none>
I20260810 01:51:42.047132 10431 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:42.047202 10431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:42.047487 10431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:42.050112 10431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "767685d9bb2e43ab8808eb268fb066a6"
format_stamp: "Formatted at 2026-08-10 01:51:42 on dist-test-slave-kvfs"
server_key: "50fa11dad3d7b6157fbf4ea14cbe1f50"
server_key_iv: "5398dfea8e56a4d0b89bfff501fb8fbb"
server_key_version: "encryptionkey@0"
I20260810 01:51:42.050854 10431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "767685d9bb2e43ab8808eb268fb066a6"
format_stamp: "Formatted at 2026-08-10 01:51:42 on dist-test-slave-kvfs"
server_key: "50fa11dad3d7b6157fbf4ea14cbe1f50"
server_key_iv: "5398dfea8e56a4d0b89bfff501fb8fbb"
server_key_version: "encryptionkey@0"
I20260810 01:51:42.055567 10431 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260810 01:51:42.058497 10457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:42.059970 10431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 01:51:42.060146 10431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "767685d9bb2e43ab8808eb268fb066a6"
format_stamp: "Formatted at 2026-08-10 01:51:42 on dist-test-slave-kvfs"
server_key: "50fa11dad3d7b6157fbf4ea14cbe1f50"
server_key_iv: "5398dfea8e56a4d0b89bfff501fb8fbb"
server_key_version: "encryptionkey@0"
I20260810 01:51:42.060257 10431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:42.084610 10431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:42.088444 10431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:42.088645 10431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:42.097658 10431 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:45167
I20260810 01:51:42.097692 10541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:45167 every 8 connection(s)
I20260810 01:51:42.098869 10431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:51:42.102366 10543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:42.105907 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 10431
I20260810 01:51:42.106069 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:51:42.106395 9526 external_mini_cluster.cc:1428] Setting key 7ad03bf0f9fd9c3f5595648b6694357a
I20260810 01:51:42.110046 10543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 767685d9bb2e43ab8808eb268fb066a6: Bootstrap starting.
I20260810 01:51:42.113461 10543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 767685d9bb2e43ab8808eb268fb066a6: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:42.114461 10543 log.cc:824] T 00000000000000000000000000000000 P 767685d9bb2e43ab8808eb268fb066a6: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:42.117097 10543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 767685d9bb2e43ab8808eb268fb066a6: No bootstrap required, opened a new log
Aug 10 01:51:42 dist-test-slave-kvfs krb5kdc[10411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:51:42.122114 10543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 767685d9bb2e43ab8808eb268fb066a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767685d9bb2e43ab8808eb268fb066a6" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45167 } }
I20260810 01:51:42.122409 10543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 767685d9bb2e43ab8808eb268fb066a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:42.122524 10543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 767685d9bb2e43ab8808eb268fb066a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 767685d9bb2e43ab8808eb268fb066a6, State: Initialized, Role: FOLLOWER
I20260810 01:51:42.123149 10543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 767685d9bb2e43ab8808eb268fb066a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767685d9bb2e43ab8808eb268fb066a6" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45167 } }
I20260810 01:51:42.123394 10543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 767685d9bb2e43ab8808eb268fb066a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:51:42.123487 10543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 767685d9bb2e43ab8808eb268fb066a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:51:42.123593 10543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 767685d9bb2e43ab8808eb268fb066a6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:42.125047 10543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 767685d9bb2e43ab8808eb268fb066a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767685d9bb2e43ab8808eb268fb066a6" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45167 } }
I20260810 01:51:42.125501 10543 leader_election.cc:302] T 00000000000000000000000000000000 P 767685d9bb2e43ab8808eb268fb066a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 767685d9bb2e43ab8808eb268fb066a6; no voters:
I20260810 01:51:42.125887 10546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:42.108265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36993 (local address 127.9.77.190:45167)
0810 01:51:42.108829 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:42.108852 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:42.108914 (+ 62us) server_negotiation.cc:407] Connection header received
0810 01:51:42.109841 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:42.109868 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:42.110412 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:42.110843 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:42.111670 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:42.113340 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:42.114582 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:42.115351 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:42.115545 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:42.118832 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:42.118839 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:42.118860 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:42.118899 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:42.121925 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:42.122543 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:42.122550 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:42.122557 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:42.122711 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:42.123109 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:42.123112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:42.123114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:42.123751 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:42.123986 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:42.124343 (+ 357us) server_negotiation.cc:299] Negotiation successful
0810 01:51:42.124608 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":167,"threads_started":1}
I20260810 01:51:42.125895 10543 leader_election.cc:288] T 00000000000000000000000000000000 P 767685d9bb2e43ab8808eb268fb066a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:51:42.126029 10549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 767685d9bb2e43ab8808eb268fb066a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:42.126271 10549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 767685d9bb2e43ab8808eb268fb066a6 [term 1 LEADER]: Becoming Leader. State: Replica: 767685d9bb2e43ab8808eb268fb066a6, State: Running, Role: LEADER
I20260810 01:51:42.126718 10549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 767685d9bb2e43ab8808eb268fb066a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767685d9bb2e43ab8808eb268fb066a6" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45167 } }
I20260810 01:51:42.127525 10543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 767685d9bb2e43ab8808eb268fb066a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:42.128300 10550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 767685d9bb2e43ab8808eb268fb066a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 767685d9bb2e43ab8808eb268fb066a6. Latest consensus state: current_term: 1 leader_uuid: "767685d9bb2e43ab8808eb268fb066a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767685d9bb2e43ab8808eb268fb066a6" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45167 } } }
I20260810 01:51:42.128535 10550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 767685d9bb2e43ab8808eb268fb066a6 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:42.129058 10563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:51:42.129827 9526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 01:51:42.133816 10563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:51:42.134099 10554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 767685d9bb2e43ab8808eb268fb066a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "767685d9bb2e43ab8808eb268fb066a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767685d9bb2e43ab8808eb268fb066a6" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 45167 } } }
I20260810 01:51:42.134514 10554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 767685d9bb2e43ab8808eb268fb066a6 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:42.141146 10563 catalog_manager.cc:1347] Generated new cluster ID: d232988c9a154b9698de755436a50a1d
I20260810 01:51:42.141230 10563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:51:42.146889 10546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:42.136595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57907 (local address 127.9.77.190:45167)
0810 01:51:42.136720 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:42.136725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:42.137036 (+ 311us) server_negotiation.cc:407] Connection header received
0810 01:51:42.137175 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:42.137179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:42.137237 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:42.137325 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:42.138270 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:42.139460 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:42.140417 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:42.140713 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:42.140820 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:42.142916 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:42.142921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:42.142926 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:42.142955 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:42.144899 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:42.145477 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:42.145481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:42.145483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:42.145578 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:42.145927 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:42.145930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:42.145932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:42.146159 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:42.146323 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:42.146602 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 01:51:42.146709 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 01:51:42.150780 10563 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:51:42.151815 10563 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:51:42.157344 10563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 767685d9bb2e43ab8808eb268fb066a6: Generated new TSK 0
I20260810 01:51:42.158182 10563 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:51:42.164078 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 10431
W20260810 01:51:42.186465 10409 connection.cc:535] client connection to 127.9.77.190:45167 recv error: Network error: failed to read from TLS socket (remote: 127.9.77.190:45167): Connection reset by peer (error 104)
2026-08-10T01:51:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1736 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:51:42 dist-test-slave-kvfs krb5kdc[10589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:51:42 dist-test-slave-kvfs krb5kdc[10589](info): set up 2 sockets
Aug 10 01:51:42 dist-test-slave-kvfs krb5kdc[10589](info): commencing operation
krb5kdc: starting...
W20260810 01:51:43.463126 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.261s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:51:43 dist-test-slave-kvfs krb5kdc[10589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:51:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:51:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:51:43.620391 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:43253
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:43253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:43.758265 10605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:43.758637 10605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:43.758687 10605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:43.763010 10605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:43.763096 10605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:43.763116 10605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:43.763134 10605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:43.763150 10605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:43.769022 10605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:43253
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:43253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:43.770396 10605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:43.771696 10605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:43.782054 10617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:43.782054 10615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:43.782094 10611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:43.782337 10605 server_base.cc:1034] running on GCE node
I20260810 01:51:43.783144 10605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:43.784513 10605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:43.785691 10605 hybrid_clock.cc:648] HybridClock initialized: now 1786326703785671 us; error 33 us; skew 500 ppm
Aug 10 01:51:43 dist-test-slave-kvfs krb5kdc[10589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326703, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:43.793970 10605 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:51:43.795145 10605 webserver.cc:460] Webserver started at http://127.9.77.190:40639/ using document root <none> and password file <none>
I20260810 01:51:43.795849 10605 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:43.795954 10605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:43.796293 10605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:43.798779 10605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "a65dc2ba6111470a8fbceed1e4d3dca5"
format_stamp: "Formatted at 2026-08-10 01:51:43 on dist-test-slave-kvfs"
server_key: "e7bc6985b142a077038691c62524694f"
server_key_iv: "bc102346b734623f0bc33bc8867b8408"
server_key_version: "encryptionkey@0"
I20260810 01:51:43.799453 10605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "a65dc2ba6111470a8fbceed1e4d3dca5"
format_stamp: "Formatted at 2026-08-10 01:51:43 on dist-test-slave-kvfs"
server_key: "e7bc6985b142a077038691c62524694f"
server_key_iv: "bc102346b734623f0bc33bc8867b8408"
server_key_version: "encryptionkey@0"
I20260810 01:51:43.803915 10605 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 01:51:43.806700 10630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:43.808223 10605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:51:43.808377 10605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "a65dc2ba6111470a8fbceed1e4d3dca5"
format_stamp: "Formatted at 2026-08-10 01:51:43 on dist-test-slave-kvfs"
server_key: "e7bc6985b142a077038691c62524694f"
server_key_iv: "bc102346b734623f0bc33bc8867b8408"
server_key_version: "encryptionkey@0"
I20260810 01:51:43.808506 10605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:43.824656 10605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:43.828764 10605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:43.829032 10605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:43.838267 10605 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:43253
I20260810 01:51:43.838299 10708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:43253 every 8 connection(s)
I20260810 01:51:43.839555 10605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:51:43.840481 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 10605
I20260810 01:51:43.840663 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:51:43.841090 9526 external_mini_cluster.cc:1428] Setting key cd9643af9b688a5d29acbbec0f0e4365
I20260810 01:51:43.844065 10709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:43.852263 10714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:43.843028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49027 (local address 127.9.77.190:43253)
0810 01:51:43.843553 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:43.843574 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:43.843628 (+ 54us) server_negotiation.cc:407] Connection header received
0810 01:51:43.844559 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:43.844584 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:43.845028 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:43.845371 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:43.845762 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:43.846866 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:43.848197 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:43.848755 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:43.848905 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:43.850022 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:43.850028 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:43.850044 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:43.850517 (+ 473us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:43.850585 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:43.850610 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:43.850724 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:43.851059 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 01:51:43.851244 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":191,"threads_started":1}
I20260810 01:51:43.852238 10709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a65dc2ba6111470a8fbceed1e4d3dca5: Bootstrap starting.
I20260810 01:51:43.855538 10709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a65dc2ba6111470a8fbceed1e4d3dca5: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:43.856001 9526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 01:51:43.856685 10709 log.cc:824] T 00000000000000000000000000000000 P a65dc2ba6111470a8fbceed1e4d3dca5: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:43.859702 10709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a65dc2ba6111470a8fbceed1e4d3dca5: No bootstrap required, opened a new log
I20260810 01:51:43.863605 10714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:43.858841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53209 (local address 127.9.77.190:43253)
0810 01:51:43.858990 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:43.858995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:43.859210 (+ 215us) server_negotiation.cc:407] Connection header received
0810 01:51:43.859339 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:43.859342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:43.859393 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:43.859495 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:43.859857 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:43.860716 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:43.861633 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:43.862017 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:43.862151 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:43.863118 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:43.863123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:43.863126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:43.863140 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:43.863175 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:43.863188 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:43.863282 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:43.863398 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 01:51:43.863448 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 01:51:43.865132 10709 raft_consensus.cc:348] T 00000000000000000000000000000000 P a65dc2ba6111470a8fbceed1e4d3dca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65dc2ba6111470a8fbceed1e4d3dca5" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 43253 } }
I20260810 01:51:43.865413 10709 raft_consensus.cc:374] T 00000000000000000000000000000000 P a65dc2ba6111470a8fbceed1e4d3dca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:43.865454 10709 raft_consensus.cc:729] T 00000000000000000000000000000000 P a65dc2ba6111470a8fbceed1e4d3dca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a65dc2ba6111470a8fbceed1e4d3dca5, State: Initialized, Role: FOLLOWER
I20260810 01:51:43.865906 10709 consensus_queue.cc:260] T 00000000000000000000000000000000 P a65dc2ba6111470a8fbceed1e4d3dca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65dc2ba6111470a8fbceed1e4d3dca5" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 43253 } }
I20260810 01:51:43.866051 10709 raft_consensus.cc:388] T 00000000000000000000000000000000 P a65dc2ba6111470a8fbceed1e4d3dca5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:51:43.866096 10709 raft_consensus.cc:482] T 00000000000000000000000000000000 P a65dc2ba6111470a8fbceed1e4d3dca5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:51:43.866178 10709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a65dc2ba6111470a8fbceed1e4d3dca5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:43.867273 10709 raft_consensus.cc:504] T 00000000000000000000000000000000 P a65dc2ba6111470a8fbceed1e4d3dca5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65dc2ba6111470a8fbceed1e4d3dca5" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 43253 } }
I20260810 01:51:43.867663 10709 leader_election.cc:302] T 00000000000000000000000000000000 P a65dc2ba6111470a8fbceed1e4d3dca5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a65dc2ba6111470a8fbceed1e4d3dca5; no voters:
I20260810 01:51:43.868013 10709 leader_election.cc:288] T 00000000000000000000000000000000 P a65dc2ba6111470a8fbceed1e4d3dca5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:51:43.868192 10727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a65dc2ba6111470a8fbceed1e4d3dca5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:43.868449 10727 raft_consensus.cc:686] T 00000000000000000000000000000000 P a65dc2ba6111470a8fbceed1e4d3dca5 [term 1 LEADER]: Becoming Leader. State: Replica: a65dc2ba6111470a8fbceed1e4d3dca5, State: Running, Role: LEADER
I20260810 01:51:43.868856 10727 consensus_queue.cc:237] T 00000000000000000000000000000000 P a65dc2ba6111470a8fbceed1e4d3dca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65dc2ba6111470a8fbceed1e4d3dca5" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 43253 } }
I20260810 01:51:43.869617 10709 sys_catalog.cc:564] T 00000000000000000000000000000000 P a65dc2ba6111470a8fbceed1e4d3dca5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:43.871016 10732 sys_catalog.cc:455] T 00000000000000000000000000000000 P a65dc2ba6111470a8fbceed1e4d3dca5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a65dc2ba6111470a8fbceed1e4d3dca5. Latest consensus state: current_term: 1 leader_uuid: "a65dc2ba6111470a8fbceed1e4d3dca5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65dc2ba6111470a8fbceed1e4d3dca5" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 43253 } } }
I20260810 01:51:43.871045 10729 sys_catalog.cc:455] T 00000000000000000000000000000000 P a65dc2ba6111470a8fbceed1e4d3dca5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a65dc2ba6111470a8fbceed1e4d3dca5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a65dc2ba6111470a8fbceed1e4d3dca5" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 43253 } } }
I20260810 01:51:43.871162 10732 sys_catalog.cc:458] T 00000000000000000000000000000000 P a65dc2ba6111470a8fbceed1e4d3dca5 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:43.871168 10729 sys_catalog.cc:458] T 00000000000000000000000000000000 P a65dc2ba6111470a8fbceed1e4d3dca5 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:43.871609 10741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:51:43.875543 10741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:51:43.882589 10741 catalog_manager.cc:1347] Generated new cluster ID: c1b2333374a04600823141d174c00ccf
I20260810 01:51:43.882685 10741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:51:43.902221 10741 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:51:43.903875 10741 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:51:43.916101 10741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a65dc2ba6111470a8fbceed1e4d3dca5: Generated new TSK 0
I20260810 01:51:43.917346 10741 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:51:43.933977 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 10605
W20260810 01:51:43.946235 10587 connection.cc:535] client connection to 127.9.77.190:43253 recv error: Network error: failed to read from TLS socket (remote: 127.9.77.190:43253): Connection reset by peer (error 104)
2026-08-10T01:51:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1759 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:51:44 dist-test-slave-kvfs krb5kdc[10755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:51:44 dist-test-slave-kvfs krb5kdc[10755](info): set up 2 sockets
Aug 10 01:51:44 dist-test-slave-kvfs krb5kdc[10755](info): commencing operation
krb5kdc: starting...
W20260810 01:51:45.245241 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.284s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:51:45 dist-test-slave-kvfs krb5kdc[10755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:51:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:51:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:51:45.400796 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:46625
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:35141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:46625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:45.540584 10769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:45.540995 10769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:45.541081 10769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:45.545439 10769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:45.545554 10769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:45.545593 10769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:45.545636 10769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:45.545678 10769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:45.551731 10769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:35141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:46625
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:46625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:45.553269 10769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:45.554586 10769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:45.564954 10776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:45.565011 10778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:45.565006 10774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:45.565264 10769 server_base.cc:1034] running on GCE node
I20260810 01:51:45.566120 10769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:45.567590 10769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:45.568817 10769 hybrid_clock.cc:648] HybridClock initialized: now 1786326705568798 us; error 29 us; skew 500 ppm
Aug 10 01:51:45 dist-test-slave-kvfs krb5kdc[10755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326705, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:45.572490 10769 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:51:45.573678 10769 webserver.cc:460] Webserver started at http://127.9.77.190:34841/ using document root <none> and password file <none>
I20260810 01:51:45.574458 10769 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:45.574565 10769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:45.574867 10769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:45.577701 10769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "3c12d5ae5d904d049c159ba3887012c7"
format_stamp: "Formatted at 2026-08-10 01:51:45 on dist-test-slave-kvfs"
server_key: "0c98977453ac6aa33f62fca3a3bbcea8"
server_key_iv: "2211ad4f58e5d91233ad5f283cce3181"
server_key_version: "encryptionkey@0"
I20260810 01:51:45.578449 10769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "3c12d5ae5d904d049c159ba3887012c7"
format_stamp: "Formatted at 2026-08-10 01:51:45 on dist-test-slave-kvfs"
server_key: "0c98977453ac6aa33f62fca3a3bbcea8"
server_key_iv: "2211ad4f58e5d91233ad5f283cce3181"
server_key_version: "encryptionkey@0"
I20260810 01:51:45.583487 10769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260810 01:51:45.587142 10792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:45.588856 10769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:51:45.589052 10769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "3c12d5ae5d904d049c159ba3887012c7"
format_stamp: "Formatted at 2026-08-10 01:51:45 on dist-test-slave-kvfs"
server_key: "0c98977453ac6aa33f62fca3a3bbcea8"
server_key_iv: "2211ad4f58e5d91233ad5f283cce3181"
server_key_version: "encryptionkey@0"
I20260810 01:51:45.589210 10769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:45.595273 10769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:45.599028 10769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:45.599267 10769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:45.608410 10769 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:46625
I20260810 01:51:45.608443 10872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:46625 every 8 connection(s)
I20260810 01:51:45.609654 10769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:51:45.610904 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 10769
I20260810 01:51:45.611089 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:51:45.611474 9526 external_mini_cluster.cc:1428] Setting key 26b2bd5e798640891548d6898991e482
I20260810 01:51:45.614136 10875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:45.618144 10879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:45.613416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33637 (local address 127.9.77.190:46625)
0810 01:51:45.613992 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:45.614011 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:45.614056 (+ 45us) server_negotiation.cc:407] Connection header received
0810 01:51:45.614906 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:45.614926 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:45.615297 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:45.615718 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:45.616703 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:45.616713 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:45.616727 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:45.616738 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:45.616805 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:45.616829 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:45.616998 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:45.617077 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 01:51:45.617267 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":198,"threads_started":1}
I20260810 01:51:45.621866 9526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 01:51:45.622433 10875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c12d5ae5d904d049c159ba3887012c7: Bootstrap starting.
I20260810 01:51:45.625733 10875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c12d5ae5d904d049c159ba3887012c7: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:45.626761 10879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:45.624546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59309 (local address 127.9.77.190:46625)
0810 01:51:45.624698 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:45.624702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:45.624990 (+ 288us) server_negotiation.cc:407] Connection header received
0810 01:51:45.625096 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:45.625099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:45.625142 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:45.625205 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:45.626201 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:45.626204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:45.626206 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:45.626217 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:45.626256 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:45.626268 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:45.626342 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:45.626505 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 01:51:45.626580 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 01:51:45.626928 10875 log.cc:824] T 00000000000000000000000000000000 P 3c12d5ae5d904d049c159ba3887012c7: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:45.629535 10875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c12d5ae5d904d049c159ba3887012c7: No bootstrap required, opened a new log
I20260810 01:51:45.633833 10875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c12d5ae5d904d049c159ba3887012c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c12d5ae5d904d049c159ba3887012c7" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46625 } }
I20260810 01:51:45.634197 10875 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c12d5ae5d904d049c159ba3887012c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:45.634298 10875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c12d5ae5d904d049c159ba3887012c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c12d5ae5d904d049c159ba3887012c7, State: Initialized, Role: FOLLOWER
I20260810 01:51:45.634879 10875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c12d5ae5d904d049c159ba3887012c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c12d5ae5d904d049c159ba3887012c7" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46625 } }
I20260810 01:51:45.635049 10875 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c12d5ae5d904d049c159ba3887012c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:51:45.635123 10875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c12d5ae5d904d049c159ba3887012c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:51:45.635254 10875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c12d5ae5d904d049c159ba3887012c7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:45.636840 10875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c12d5ae5d904d049c159ba3887012c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c12d5ae5d904d049c159ba3887012c7" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46625 } }
I20260810 01:51:45.637370 10875 leader_election.cc:302] T 00000000000000000000000000000000 P 3c12d5ae5d904d049c159ba3887012c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c12d5ae5d904d049c159ba3887012c7; no voters:
I20260810 01:51:45.637876 10875 leader_election.cc:288] T 00000000000000000000000000000000 P 3c12d5ae5d904d049c159ba3887012c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:51:45.637959 10889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c12d5ae5d904d049c159ba3887012c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:45.638207 10889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c12d5ae5d904d049c159ba3887012c7 [term 1 LEADER]: Becoming Leader. State: Replica: 3c12d5ae5d904d049c159ba3887012c7, State: Running, Role: LEADER
I20260810 01:51:45.638622 10889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c12d5ae5d904d049c159ba3887012c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c12d5ae5d904d049c159ba3887012c7" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46625 } }
I20260810 01:51:45.639608 10875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c12d5ae5d904d049c159ba3887012c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:45.640803 10891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c12d5ae5d904d049c159ba3887012c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c12d5ae5d904d049c159ba3887012c7. Latest consensus state: current_term: 1 leader_uuid: "3c12d5ae5d904d049c159ba3887012c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c12d5ae5d904d049c159ba3887012c7" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46625 } } }
I20260810 01:51:45.640961 10891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c12d5ae5d904d049c159ba3887012c7 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:45.641366 10890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c12d5ae5d904d049c159ba3887012c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c12d5ae5d904d049c159ba3887012c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c12d5ae5d904d049c159ba3887012c7" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 46625 } } }
I20260810 01:51:45.641515 10890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c12d5ae5d904d049c159ba3887012c7 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:45.642441 10907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:51:45.645972 10907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:51:45.653395 10907 catalog_manager.cc:1347] Generated new cluster ID: d079db57793a4373af1136ce12143784
I20260810 01:51:45.653504 10907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:51:45.665782 10907 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:51:45.666231 10907 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:51:45.682713 10907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c12d5ae5d904d049c159ba3887012c7: Generated new TSK 0
I20260810 01:51:45.683877 10907 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:51:45.693982 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 10769
2026-08-10T01:51:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1765 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:51:45 dist-test-slave-kvfs krb5kdc[10922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:51:45 dist-test-slave-kvfs krb5kdc[10922](info): set up 2 sockets
Aug 10 01:51:45 dist-test-slave-kvfs krb5kdc[10922](info): commencing operation
krb5kdc: starting...
W20260810 01:51:47.003724 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.276s user 0.001s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:51:47 dist-test-slave-kvfs krb5kdc[10922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:51:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:51:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:51:47.155805 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:39273
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:36005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:39273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:47.306015 10943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:47.306370 10943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:47.306416 10943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:47.310631 10943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:47.310708 10943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:47.310730 10943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:47.310755 10943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:47.310771 10943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:47.316798 10943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:36005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:39273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:39273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:47.318171 10943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:47.319409 10943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:47.329996 10951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:47.329996 10955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:47.329989 10952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:47.330368 10943 server_base.cc:1034] running on GCE node
I20260810 01:51:47.331192 10943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:47.332758 10943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:47.333973 10943 hybrid_clock.cc:648] HybridClock initialized: now 1786326707333952 us; error 36 us; skew 500 ppm
Aug 10 01:51:47 dist-test-slave-kvfs krb5kdc[10922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326707, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:47.337484 10943 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:51:47.338730 10943 webserver.cc:460] Webserver started at http://127.9.77.190:36455/ using document root <none> and password file <none>
I20260810 01:51:47.339466 10943 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:47.339546 10943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:47.339767 10943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:47.342417 10943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "1bf86378100e40bc8be44b6f00ca553b"
format_stamp: "Formatted at 2026-08-10 01:51:47 on dist-test-slave-kvfs"
server_key: "b0368510f83889cda654ef835d3ad0a5"
server_key_iv: "f6c45d5122c262c9e61adac18339644b"
server_key_version: "encryptionkey@0"
I20260810 01:51:47.343039 10943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "1bf86378100e40bc8be44b6f00ca553b"
format_stamp: "Formatted at 2026-08-10 01:51:47 on dist-test-slave-kvfs"
server_key: "b0368510f83889cda654ef835d3ad0a5"
server_key_iv: "f6c45d5122c262c9e61adac18339644b"
server_key_version: "encryptionkey@0"
I20260810 01:51:47.347787 10943 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 01:51:47.350771 10967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:47.352533 10943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:51:47.352700 10943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "1bf86378100e40bc8be44b6f00ca553b"
format_stamp: "Formatted at 2026-08-10 01:51:47 on dist-test-slave-kvfs"
server_key: "b0368510f83889cda654ef835d3ad0a5"
server_key_iv: "f6c45d5122c262c9e61adac18339644b"
server_key_version: "encryptionkey@0"
I20260810 01:51:47.352820 10943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:47.369788 10943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:47.373587 10943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:47.373816 10943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:47.382315 10943 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:39273
I20260810 01:51:47.382334 11040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:39273 every 8 connection(s)
I20260810 01:51:47.383489 10943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:51:47.387058 11041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:47.387310 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 10943
I20260810 01:51:47.387490 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:51:47.387890 9526 external_mini_cluster.cc:1428] Setting key 9a1caf3ad212a3e78c7ec5a97710fa8f
I20260810 01:51:47.394963 11041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bf86378100e40bc8be44b6f00ca553b: Bootstrap starting.
I20260810 01:51:47.397933 11041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bf86378100e40bc8be44b6f00ca553b: Neither blocks nor log segments found. Creating new log.
Aug 10 01:51:47 dist-test-slave-kvfs krb5kdc[10922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:51:47.399039 11041 log.cc:824] T 00000000000000000000000000000000 P 1bf86378100e40bc8be44b6f00ca553b: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:47.401669 11041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bf86378100e40bc8be44b6f00ca553b: No bootstrap required, opened a new log
I20260810 01:51:47.405042 11045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:47.389691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60035 (local address 127.9.77.190:39273)
0810 01:51:47.390272 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:47.390291 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:47.390338 (+ 47us) server_negotiation.cc:407] Connection header received
0810 01:51:47.391190 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:47.391213 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:47.391653 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:47.392142 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:47.393165 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:47.394367 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:47.395490 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:47.396158 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:47.396367 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:47.399687 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:47.399693 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:47.399710 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:47.399742 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:47.401655 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:47.402214 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:47.402220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:47.402226 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:47.402364 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:47.402678 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:47.402682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:47.402684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:47.403157 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:47.403415 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:47.403701 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 01:51:47.404130 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":198,"threads_started":1}
I20260810 01:51:47.406147 11041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bf86378100e40bc8be44b6f00ca553b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bf86378100e40bc8be44b6f00ca553b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 39273 } }
I20260810 01:51:47.406417 11041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bf86378100e40bc8be44b6f00ca553b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:47.406497 11041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bf86378100e40bc8be44b6f00ca553b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bf86378100e40bc8be44b6f00ca553b, State: Initialized, Role: FOLLOWER
I20260810 01:51:47.407014 11041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bf86378100e40bc8be44b6f00ca553b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bf86378100e40bc8be44b6f00ca553b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 39273 } }
I20260810 01:51:47.407197 11041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bf86378100e40bc8be44b6f00ca553b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:51:47.407289 11041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bf86378100e40bc8be44b6f00ca553b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:51:47.407418 11041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bf86378100e40bc8be44b6f00ca553b [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:47.408545 9526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 01:51:47.408705 11041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bf86378100e40bc8be44b6f00ca553b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bf86378100e40bc8be44b6f00ca553b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 39273 } }
I20260810 01:51:47.409106 11041 leader_election.cc:302] T 00000000000000000000000000000000 P 1bf86378100e40bc8be44b6f00ca553b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bf86378100e40bc8be44b6f00ca553b; no voters:
I20260810 01:51:47.409515 11041 leader_election.cc:288] T 00000000000000000000000000000000 P 1bf86378100e40bc8be44b6f00ca553b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:51:47.409605 11050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bf86378100e40bc8be44b6f00ca553b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:47.409823 11050 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bf86378100e40bc8be44b6f00ca553b [term 1 LEADER]: Becoming Leader. State: Replica: 1bf86378100e40bc8be44b6f00ca553b, State: Running, Role: LEADER
I20260810 01:51:47.410234 11050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bf86378100e40bc8be44b6f00ca553b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bf86378100e40bc8be44b6f00ca553b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 39273 } }
I20260810 01:51:47.411077 11041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bf86378100e40bc8be44b6f00ca553b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:47.411715 11050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bf86378100e40bc8be44b6f00ca553b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bf86378100e40bc8be44b6f00ca553b. Latest consensus state: current_term: 1 leader_uuid: "1bf86378100e40bc8be44b6f00ca553b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bf86378100e40bc8be44b6f00ca553b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 39273 } } }
I20260810 01:51:47.411923 11050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bf86378100e40bc8be44b6f00ca553b [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:47.412264 11051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bf86378100e40bc8be44b6f00ca553b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bf86378100e40bc8be44b6f00ca553b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bf86378100e40bc8be44b6f00ca553b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 39273 } } }
I20260810 01:51:47.412379 11051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bf86378100e40bc8be44b6f00ca553b [sys.catalog]: This master's current role is: LEADER
W20260810 01:51:47.417207 11082 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1bf86378100e40bc8be44b6f00ca553b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 01:51:47.417341 11082 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 01:51:47.417701 11069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:51:47.418680 11069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:51:47.422667 11045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:47.412710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49579 (local address 127.9.77.190:39273)
0810 01:51:47.412837 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:47.412842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:47.413227 (+ 385us) server_negotiation.cc:407] Connection header received
0810 01:51:47.413417 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:47.413420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:47.413473 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:47.413538 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:47.414705 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:47.415535 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:47.416781 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:47.417136 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:47.417255 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:47.418980 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:47.418984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:47.418987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:47.419013 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:47.420612 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:47.421154 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:47.421160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:47.421163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:47.421287 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:47.421615 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:47.421618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:47.421620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:47.421874 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:47.422031 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:47.422312 (+ 281us) server_negotiation.cc:299] Negotiation successful
0810 01:51:47.422465 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 01:51:47.425848 11069 catalog_manager.cc:1347] Generated new cluster ID: acb5c63b6b60413db04826acba51e192
I20260810 01:51:47.425923 11069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:51:47.436259 11069 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:51:47.437708 11069 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:51:47.444785 11069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bf86378100e40bc8be44b6f00ca553b: Generated new TSK 0
I20260810 01:51:47.445811 11069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:51:47.449918 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 10943
2026-08-10T01:51:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1752 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:51:47 dist-test-slave-kvfs krb5kdc[11093](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:51:47 dist-test-slave-kvfs krb5kdc[11093](info): set up 2 sockets
Aug 10 01:51:47 dist-test-slave-kvfs krb5kdc[11093](info): commencing operation
krb5kdc: starting...
W20260810 01:51:48.755373 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.275s user 0.007s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:51:48 dist-test-slave-kvfs krb5kdc[11093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:51:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:51:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:51:48.911084 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:34023
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:38965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:34023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:49.050452 11108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:49.050866 11108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:49.050956 11108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:49.055238 11108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:49.055352 11108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:49.055389 11108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:49.055426 11108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:49.055485 11108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:49.061287 11108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:38965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:34023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:34023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:49.062714 11108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:49.064116 11108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:49.074236 11122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:49.074237 11124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:49.074442 11108 server_base.cc:1034] running on GCE node
W20260810 01:51:49.074442 11128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:49.075369 11108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:49.076813 11108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:49.078018 11108 hybrid_clock.cc:648] HybridClock initialized: now 1786326709077998 us; error 35 us; skew 500 ppm
Aug 10 01:51:49 dist-test-slave-kvfs krb5kdc[11093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326709, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:49.086993 11108 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:51:49.088285 11108 webserver.cc:460] Webserver started at http://127.9.77.190:44983/ using document root <none> and password file <none>
I20260810 01:51:49.089027 11108 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:49.089125 11108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:49.089422 11108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:49.092092 11108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "e213f039b10346aa8d43df8c3dd02c3b"
format_stamp: "Formatted at 2026-08-10 01:51:49 on dist-test-slave-kvfs"
server_key: "d4c64e27c103bcee2673d8bbfdd8dcc8"
server_key_iv: "3a8b578e6ca59cca1411614d61854442"
server_key_version: "encryptionkey@0"
I20260810 01:51:49.092801 11108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "e213f039b10346aa8d43df8c3dd02c3b"
format_stamp: "Formatted at 2026-08-10 01:51:49 on dist-test-slave-kvfs"
server_key: "d4c64e27c103bcee2673d8bbfdd8dcc8"
server_key_iv: "3a8b578e6ca59cca1411614d61854442"
server_key_version: "encryptionkey@0"
I20260810 01:51:49.097663 11108 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260810 01:51:49.101042 11136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:49.102789 11108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 01:51:49.102995 11108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "e213f039b10346aa8d43df8c3dd02c3b"
format_stamp: "Formatted at 2026-08-10 01:51:49 on dist-test-slave-kvfs"
server_key: "d4c64e27c103bcee2673d8bbfdd8dcc8"
server_key_iv: "3a8b578e6ca59cca1411614d61854442"
server_key_version: "encryptionkey@0"
I20260810 01:51:49.103161 11108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:49.117350 11108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:49.121266 11108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:49.121646 11108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:49.131068 11108 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:34023
I20260810 01:51:49.131057 11218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:34023 every 8 connection(s)
I20260810 01:51:49.132414 11108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:51:49.136440 11219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:49.141383 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 11108
I20260810 01:51:49.141566 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:51:49.141968 9526 external_mini_cluster.cc:1428] Setting key feec640deb2996c40c59f291d7f2f6e2
I20260810 01:51:49.143906 11219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e213f039b10346aa8d43df8c3dd02c3b: Bootstrap starting.
I20260810 01:51:49.147030 11219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e213f039b10346aa8d43df8c3dd02c3b: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:49.148252 11219 log.cc:824] T 00000000000000000000000000000000 P e213f039b10346aa8d43df8c3dd02c3b: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:49.151139 11219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e213f039b10346aa8d43df8c3dd02c3b: No bootstrap required, opened a new log
I20260810 01:51:49.152990 11223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:49.143663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54443 (local address 127.9.77.190:34023)
0810 01:51:49.144188 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:49.144223 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:49.144290 (+ 67us) server_negotiation.cc:407] Connection header received
0810 01:51:49.145063 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:49.145083 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:49.145492 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:49.145822 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:49.146214 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:49.147429 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:49.148819 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:49.149576 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:49.149723 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:49.150785 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:49.150790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:49.150807 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:49.151365 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:49.151436 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:49.151463 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:49.151551 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:49.151742 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 01:51:49.152030 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":161,"threads_started":1}
I20260810 01:51:49.155571 11219 raft_consensus.cc:348] T 00000000000000000000000000000000 P e213f039b10346aa8d43df8c3dd02c3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e213f039b10346aa8d43df8c3dd02c3b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 34023 } }
I20260810 01:51:49.155872 11219 raft_consensus.cc:374] T 00000000000000000000000000000000 P e213f039b10346aa8d43df8c3dd02c3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:49.155959 11219 raft_consensus.cc:729] T 00000000000000000000000000000000 P e213f039b10346aa8d43df8c3dd02c3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e213f039b10346aa8d43df8c3dd02c3b, State: Initialized, Role: FOLLOWER
I20260810 01:51:49.156430 9526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 01:51:49.156512 11219 consensus_queue.cc:260] T 00000000000000000000000000000000 P e213f039b10346aa8d43df8c3dd02c3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e213f039b10346aa8d43df8c3dd02c3b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 34023 } }
I20260810 01:51:49.156728 11219 raft_consensus.cc:388] T 00000000000000000000000000000000 P e213f039b10346aa8d43df8c3dd02c3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:51:49.156821 11219 raft_consensus.cc:482] T 00000000000000000000000000000000 P e213f039b10346aa8d43df8c3dd02c3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:51:49.156960 11219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e213f039b10346aa8d43df8c3dd02c3b [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:49.158159 11219 raft_consensus.cc:504] T 00000000000000000000000000000000 P e213f039b10346aa8d43df8c3dd02c3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e213f039b10346aa8d43df8c3dd02c3b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 34023 } }
I20260810 01:51:49.158571 11219 leader_election.cc:302] T 00000000000000000000000000000000 P e213f039b10346aa8d43df8c3dd02c3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e213f039b10346aa8d43df8c3dd02c3b; no voters:
I20260810 01:51:49.158977 11219 leader_election.cc:288] T 00000000000000000000000000000000 P e213f039b10346aa8d43df8c3dd02c3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:51:49.159075 11231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e213f039b10346aa8d43df8c3dd02c3b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:49.159628 11231 raft_consensus.cc:686] T 00000000000000000000000000000000 P e213f039b10346aa8d43df8c3dd02c3b [term 1 LEADER]: Becoming Leader. State: Replica: e213f039b10346aa8d43df8c3dd02c3b, State: Running, Role: LEADER
I20260810 01:51:49.159987 11231 consensus_queue.cc:237] T 00000000000000000000000000000000 P e213f039b10346aa8d43df8c3dd02c3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e213f039b10346aa8d43df8c3dd02c3b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 34023 } }
I20260810 01:51:49.160672 11219 sys_catalog.cc:564] T 00000000000000000000000000000000 P e213f039b10346aa8d43df8c3dd02c3b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:49.162009 11236 sys_catalog.cc:455] T 00000000000000000000000000000000 P e213f039b10346aa8d43df8c3dd02c3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e213f039b10346aa8d43df8c3dd02c3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e213f039b10346aa8d43df8c3dd02c3b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 34023 } } }
I20260810 01:51:49.162170 11236 sys_catalog.cc:458] T 00000000000000000000000000000000 P e213f039b10346aa8d43df8c3dd02c3b [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:49.162568 11237 sys_catalog.cc:455] T 00000000000000000000000000000000 P e213f039b10346aa8d43df8c3dd02c3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e213f039b10346aa8d43df8c3dd02c3b. Latest consensus state: current_term: 1 leader_uuid: "e213f039b10346aa8d43df8c3dd02c3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e213f039b10346aa8d43df8c3dd02c3b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 34023 } } }
I20260810 01:51:49.162724 11237 sys_catalog.cc:458] T 00000000000000000000000000000000 P e213f039b10346aa8d43df8c3dd02c3b [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:49.164381 11223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:49.159116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52955 (local address 127.9.77.190:34023)
0810 01:51:49.159285 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:49.159290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:49.159616 (+ 326us) server_negotiation.cc:407] Connection header received
0810 01:51:49.159796 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:49.159801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:49.159861 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:49.159948 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:49.160233 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:49.160977 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:49.162158 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:49.162497 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:49.162595 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:49.163885 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:49.163888 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:49.163890 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:49.163903 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:49.163928 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:49.163939 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:49.164005 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:49.164164 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 01:51:49.164234 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260810 01:51:49.166899 11258 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e213f039b10346aa8d43df8c3dd02c3b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 01:51:49.166972 11258 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 01:51:49.167060 11248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:51:49.167851 11248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:51:49.174760 11248 catalog_manager.cc:1347] Generated new cluster ID: 08cce8ed4e4b4752aae57d3d5921abca
I20260810 01:51:49.174856 11248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:51:49.188418 11248 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:51:49.189604 11248 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:51:49.199790 11248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e213f039b10346aa8d43df8c3dd02c3b: Generated new TSK 0
I20260810 01:51:49.200811 11248 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:51:49.210597 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 11108
2026-08-10T01:51:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1768 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:51:49 dist-test-slave-kvfs krb5kdc[11270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:51:49 dist-test-slave-kvfs krb5kdc[11270](info): set up 2 sockets
Aug 10 01:51:49 dist-test-slave-kvfs krb5kdc[11270](info): commencing operation
krb5kdc: starting...
W20260810 01:51:50.539940 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.291s user 0.005s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:51:50 dist-test-slave-kvfs krb5kdc[11270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:51:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:51:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:51:50.700201 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:43679
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:43143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:43679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:50.837576 11290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:50.837998 11290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:50.838083 11290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:50.842469 11290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:50.842587 11290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:50.842625 11290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:50.842669 11290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:50.842710 11290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:50.848742 11290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:43143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:43679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:43679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:50.850467 11290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:50.851929 11290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:50.862886 11308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:50.863104 11290 server_base.cc:1034] running on GCE node
W20260810 01:51:50.862890 11301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:50.862890 11303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:50.863974 11290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:50.865536 11290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:50.866746 11290 hybrid_clock.cc:648] HybridClock initialized: now 1786326710866698 us; error 62 us; skew 500 ppm
Aug 10 01:51:50 dist-test-slave-kvfs krb5kdc[11270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326710, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:50.870344 11290 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:51:50.871485 11290 webserver.cc:460] Webserver started at http://127.9.77.190:42249/ using document root <none> and password file <none>
I20260810 01:51:50.872267 11290 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:50.872372 11290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:50.872669 11290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:50.875237 11290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "f74911d77d9c466ab58ecf854c3b3a93"
format_stamp: "Formatted at 2026-08-10 01:51:50 on dist-test-slave-kvfs"
server_key: "ce7b055d95bc9b235d3cd4c5d69dbbb6"
server_key_iv: "632f3e8e380060f29af0786a38d1efe9"
server_key_version: "encryptionkey@0"
I20260810 01:51:50.875928 11290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "f74911d77d9c466ab58ecf854c3b3a93"
format_stamp: "Formatted at 2026-08-10 01:51:50 on dist-test-slave-kvfs"
server_key: "ce7b055d95bc9b235d3cd4c5d69dbbb6"
server_key_iv: "632f3e8e380060f29af0786a38d1efe9"
server_key_version: "encryptionkey@0"
I20260810 01:51:50.880721 11290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 01:51:50.883948 11321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:50.885646 11290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:51:50.885847 11290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "f74911d77d9c466ab58ecf854c3b3a93"
format_stamp: "Formatted at 2026-08-10 01:51:50 on dist-test-slave-kvfs"
server_key: "ce7b055d95bc9b235d3cd4c5d69dbbb6"
server_key_iv: "632f3e8e380060f29af0786a38d1efe9"
server_key_version: "encryptionkey@0"
I20260810 01:51:50.886005 11290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:50.891966 11290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:50.896128 11290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:50.896376 11290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:50.905637 11290 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:43679
I20260810 01:51:50.905679 11392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:43679 every 8 connection(s)
I20260810 01:51:50.906975 11290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:51:50.910504 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 11290
I20260810 01:51:50.910670 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:51:50.910776 11397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:50.911054 9526 external_mini_cluster.cc:1428] Setting key e4512f77bf96b1097716feeffcb7919c
I20260810 01:51:50.918056 11400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:50.912931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39099 (local address 127.9.77.190:43679)
0810 01:51:50.913489 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:50.913512 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:50.913568 (+ 56us) server_negotiation.cc:407] Connection header received
0810 01:51:50.914334 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:50.914364 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:50.914837 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:50.915299 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:50.916254 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:50.916261 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:50.916279 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:50.916292 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:50.916372 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:50.916401 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:50.916575 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:50.916687 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 01:51:50.916883 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":158,"threads_started":1}
I20260810 01:51:50.920081 11397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f74911d77d9c466ab58ecf854c3b3a93: Bootstrap starting.
I20260810 01:51:50.921753 9526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 01:51:50.923367 11397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f74911d77d9c466ab58ecf854c3b3a93: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:50.924448 11397 log.cc:824] T 00000000000000000000000000000000 P f74911d77d9c466ab58ecf854c3b3a93: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:50.926570 11400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:50.924459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33103 (local address 127.9.77.190:43679)
0810 01:51:50.924626 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:50.924632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:50.924946 (+ 314us) server_negotiation.cc:407] Connection header received
0810 01:51:50.925051 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:50.925054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:50.925100 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:50.925207 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:50.926078 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:50.926082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:50.926084 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 01:51:50.926095 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:50.926124 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 01:51:50.926135 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:50.926219 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:50.926322 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 01:51:50.926361 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 01:51:50.927527 11397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f74911d77d9c466ab58ecf854c3b3a93: No bootstrap required, opened a new log
I20260810 01:51:50.932309 11397 raft_consensus.cc:348] T 00000000000000000000000000000000 P f74911d77d9c466ab58ecf854c3b3a93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74911d77d9c466ab58ecf854c3b3a93" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 43679 } }
I20260810 01:51:50.932708 11397 raft_consensus.cc:374] T 00000000000000000000000000000000 P f74911d77d9c466ab58ecf854c3b3a93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 01:51:50.932806 11397 raft_consensus.cc:729] T 00000000000000000000000000000000 P f74911d77d9c466ab58ecf854c3b3a93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74911d77d9c466ab58ecf854c3b3a93, State: Initialized, Role: FOLLOWER
I20260810 01:51:50.933450 11397 consensus_queue.cc:260] T 00000000000000000000000000000000 P f74911d77d9c466ab58ecf854c3b3a93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74911d77d9c466ab58ecf854c3b3a93" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 43679 } }
I20260810 01:51:50.933647 11397 raft_consensus.cc:388] T 00000000000000000000000000000000 P f74911d77d9c466ab58ecf854c3b3a93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 01:51:50.933748 11397 raft_consensus.cc:482] T 00000000000000000000000000000000 P f74911d77d9c466ab58ecf854c3b3a93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 01:51:50.933904 11397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f74911d77d9c466ab58ecf854c3b3a93 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:50.935226 11397 raft_consensus.cc:504] T 00000000000000000000000000000000 P f74911d77d9c466ab58ecf854c3b3a93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74911d77d9c466ab58ecf854c3b3a93" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 43679 } }
I20260810 01:51:50.935700 11397 leader_election.cc:302] T 00000000000000000000000000000000 P f74911d77d9c466ab58ecf854c3b3a93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f74911d77d9c466ab58ecf854c3b3a93; no voters:
I20260810 01:51:50.936237 11397 leader_election.cc:288] T 00000000000000000000000000000000 P f74911d77d9c466ab58ecf854c3b3a93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 01:51:50.936401 11410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f74911d77d9c466ab58ecf854c3b3a93 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:50.936753 11410 raft_consensus.cc:686] T 00000000000000000000000000000000 P f74911d77d9c466ab58ecf854c3b3a93 [term 1 LEADER]: Becoming Leader. State: Replica: f74911d77d9c466ab58ecf854c3b3a93, State: Running, Role: LEADER
I20260810 01:51:50.937233 11410 consensus_queue.cc:237] T 00000000000000000000000000000000 P f74911d77d9c466ab58ecf854c3b3a93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74911d77d9c466ab58ecf854c3b3a93" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 43679 } }
I20260810 01:51:50.938011 11397 sys_catalog.cc:564] T 00000000000000000000000000000000 P f74911d77d9c466ab58ecf854c3b3a93 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:50.939673 11412 sys_catalog.cc:455] T 00000000000000000000000000000000 P f74911d77d9c466ab58ecf854c3b3a93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f74911d77d9c466ab58ecf854c3b3a93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74911d77d9c466ab58ecf854c3b3a93" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 43679 } } }
I20260810 01:51:50.939828 11412 sys_catalog.cc:458] T 00000000000000000000000000000000 P f74911d77d9c466ab58ecf854c3b3a93 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:50.939802 11413 sys_catalog.cc:455] T 00000000000000000000000000000000 P f74911d77d9c466ab58ecf854c3b3a93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f74911d77d9c466ab58ecf854c3b3a93. Latest consensus state: current_term: 1 leader_uuid: "f74911d77d9c466ab58ecf854c3b3a93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74911d77d9c466ab58ecf854c3b3a93" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 43679 } } }
I20260810 01:51:50.940289 11413 sys_catalog.cc:458] T 00000000000000000000000000000000 P f74911d77d9c466ab58ecf854c3b3a93 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:50.940368 11425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:51:50.944445 11425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:51:50.952205 11425 catalog_manager.cc:1347] Generated new cluster ID: 62f942d939b44efaac0aaa399e8e5779
I20260810 01:51:50.952330 11425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:51:50.973975 11425 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:51:50.974478 11425 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:51:50.980297 11425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f74911d77d9c466ab58ecf854c3b3a93: Generated new TSK 0
I20260810 01:51:50.981258 11425 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 01:51:50.986940 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 11290
2026-08-10T01:51:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1771 ms)
[----------] 18 tests from AuthTokenIssuingTest (31964 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:51:51 dist-test-slave-kvfs krb5kdc[11441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:51:51 dist-test-slave-kvfs krb5kdc[11441](info): set up 2 sockets
Aug 10 01:51:51 dist-test-slave-kvfs krb5kdc[11441](info): commencing operation
krb5kdc: starting...
W20260810 01:51:52.290431 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.270s user 0.003s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:51:52 dist-test-slave-kvfs krb5kdc[11441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:51:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:51:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:51:52.443718 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:37443
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:34057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:37443,127.9.77.189:42129,127.9.77.188:38023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:52.583644 11461 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 01:51:52.584131 11461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:52.584214 11461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:52.584301 11461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:52.588495 11461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:52.588593 11461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:52.588629 11461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:52.588668 11461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:52.588704 11461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:52.594307 11461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:34057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:37443,127.9.77.189:42129,127.9.77.188:38023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:37443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:52.595714 11461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:52.597065 11461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:52.607532 11472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:52.607795 11461 server_base.cc:1034] running on GCE node
W20260810 01:51:52.607532 11468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:52.607544 11469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:52.608598 11461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:52.609965 11461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:52.611169 11461 hybrid_clock.cc:648] HybridClock initialized: now 1786326712611147 us; error 34 us; skew 500 ppm
Aug 10 01:51:52 dist-test-slave-kvfs krb5kdc[11441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326712, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:52.614538 11461 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:51:52.615583 11461 webserver.cc:460] Webserver started at http://127.9.77.190:33069/ using document root <none> and password file <none>
I20260810 01:51:52.616348 11461 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:52.616446 11461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:52.616734 11461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:52.619225 11461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "84d9d7a3615749b29232b0c070524599"
format_stamp: "Formatted at 2026-08-10 01:51:52 on dist-test-slave-kvfs"
server_key: "a130d4e52699e10633433357be265431"
server_key_iv: "3bcd965ccc54c3534a426d20cebce45b"
server_key_version: "encryptionkey@0"
I20260810 01:51:52.619913 11461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "84d9d7a3615749b29232b0c070524599"
format_stamp: "Formatted at 2026-08-10 01:51:52 on dist-test-slave-kvfs"
server_key: "a130d4e52699e10633433357be265431"
server_key_iv: "3bcd965ccc54c3534a426d20cebce45b"
server_key_version: "encryptionkey@0"
I20260810 01:51:52.624383 11461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 01:51:52.627308 11481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:52.628993 11461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 01:51:52.629151 11461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "84d9d7a3615749b29232b0c070524599"
format_stamp: "Formatted at 2026-08-10 01:51:52 on dist-test-slave-kvfs"
server_key: "a130d4e52699e10633433357be265431"
server_key_iv: "3bcd965ccc54c3534a426d20cebce45b"
server_key_version: "encryptionkey@0"
I20260810 01:51:52.629288 11461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:52.659108 11461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:52.663318 11461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:52.663581 11461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:52.672479 11461 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:37443
I20260810 01:51:52.672865 11559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:37443 every 8 connection(s)
I20260810 01:51:52.673705 11461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:51:52.674211 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 11461
I20260810 01:51:52.674391 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:51:52.674754 9526 external_mini_cluster.cc:1428] Setting key 8b1afecf0cb3cb2c1969197d940c7e1b
I20260810 01:51:52.677537 11562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:52.682725 11562 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37443 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.77.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.189@KRBTEST.COM" created.
Aug 10 01:51:52 dist-test-slave-kvfs krb5kdc[11441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326712, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:51:52.708070 11566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:52.686654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42630 (local address 127.9.77.190:37443)
0810 01:51:52.688544 (+ 1890us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:52.688556 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:52.688606 (+ 50us) server_negotiation.cc:407] Connection header received
0810 01:51:52.689763 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:52.689769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:52.690122 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:52.690434 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:52.691766 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:52.693506 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:52.697009 (+ 3503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:52.697791 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:52.698008 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:52.702080 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:52.702087 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:52.702103 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:52.702127 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:52.703999 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:52.704718 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:52.704724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:52.704725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:52.704868 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:52.705343 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:52.705346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:52.705347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:52.705963 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:52.706090 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:52.706634 (+ 544us) server_negotiation.cc:299] Negotiation successful
0810 01:51:52.706916 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":1713,"spinlock_wait_cycles":45312,"thread_start_us":253,"threads_started":1}
I20260810 01:51:52.708015 11565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:52.686652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37443 (local address 127.0.0.1:42630)
0810 01:51:52.687621 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:52.687688 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:52.689292 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:52.690419 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:52.690428 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:52.691087 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:52.691542 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:52.691565 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:52.694999 (+ 3434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:52.695005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:52.696791 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:52.696808 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:52.698025 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:52.698033 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:52.698229 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:52.699118 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:51:52.699151 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:52.701886 (+ 2735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:52.704161 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:52.704178 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:52.704197 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:52.704585 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:52.704981 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:52.704985 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:52.704988 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:52.705184 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:52.706074 (+ 890us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:52.706080 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:52.706248 (+ 168us) client_negotiation.cc:769] Sending connection context
0810 01:51:52.706434 (+ 186us) client_negotiation.cc:240] Negotiation successful
0810 01:51:52.706799 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":220,"threads_started":1}
I20260810 01:51:52.711135 11562 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.189" port: 42129 } has no permanent_uuid. Determining permanent_uuid...
I20260810 01:51:52.712136 11565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:52.711762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:42129 (local address 127.0.0.1:51058)
0810 01:51:52.711919 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:52.712006 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.189:42129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 01:51:52.712356 11485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.77.189:42129: connect: Connection refused (error 111)
I20260810 01:51:52.713374 11565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:52.713172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:42129 (local address 127.0.0.1:51064)
0810 01:51:52.713272 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:52.713320 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.189:42129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 01:51:52.713555 11562 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.189:42129: Network error: Client connection negotiation failed: client connection to 127.9.77.189:42129: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.189.keytab.
Entry for principal kudu/127.9.77.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.189.keytab.
I20260810 01:51:52.715557 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.189:42129
--webserver_interface=127.9.77.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.189.keytab
--principal=kudu/127.9.77.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:34057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:37443,127.9.77.189:42129,127.9.77.188:38023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 01:51:52.762926 11562 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.189" port: 42129 } attempt: 1
I20260810 01:51:52.764226 11565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:52.763735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:42129 (local address 127.0.0.1:51076)
0810 01:51:52.763964 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:52.764096 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.189:42129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260810 01:51:52.765218 11565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:52.764951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:42129 (local address 127.0.0.1:51082)
0810 01:51:52.765071 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:52.765133 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.189:42129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 01:51:52.765395 11562 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.189:42129: Network error: Client connection negotiation failed: client connection to 127.9.77.189:42129: connect: Connection refused (error 111)
I20260810 01:51:52.833645 11562 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.189" port: 42129 } attempt: 2
I20260810 01:51:52.834940 11565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:52.834442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:42129 (local address 127.0.0.1:51096)
0810 01:51:52.834671 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:52.834771 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.189:42129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260810 01:51:52.835992 11565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:52.835688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:42129 (local address 127.0.0.1:51106)
0810 01:51:52.835850 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:52.835912 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.189:42129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260810 01:51:52.836179 11562 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.189:42129: Network error: Client connection negotiation failed: client connection to 127.9.77.189:42129: connect: Connection refused (error 111)
W20260810 01:51:52.856781 11569 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 01:51:52.857196 11569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:52.857223 11569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:52.857329 11569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:52.861627 11569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:52.861704 11569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:52.861725 11569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:52.861743 11569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:52.861761 11569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:52.867769 11569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:34057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:37443,127.9.77.189:42129,127.9.77.188:38023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.189.keytab
--principal=kudu/127.9.77.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.189:42129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.77.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:52.869342 11569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:52.870718 11569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:52.881083 11584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:52.881086 11578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:52.881086 11579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:52.881140 11569 server_base.cc:1034] running on GCE node
I20260810 01:51:52.882287 11569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:52.883944 11569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:52.885233 11569 hybrid_clock.cc:648] HybridClock initialized: now 1786326712885186 us; error 61 us; skew 500 ppm
Aug 10 01:51:52 dist-test-slave-kvfs krb5kdc[11441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326712, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:52.888837 11569 init.cc:377] Logged in from keytab as kudu/127.9.77.189@KRBTEST.COM (short username kudu)
I20260810 01:51:52.890018 11569 webserver.cc:460] Webserver started at http://127.9.77.189:38247/ using document root <none> and password file <none>
I20260810 01:51:52.890720 11569 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:52.890793 11569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:52.891008 11569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:52.893582 11569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-1/data/instance:
uuid: "9b061833a054483b82f3c6f79cce2414"
format_stamp: "Formatted at 2026-08-10 01:51:52 on dist-test-slave-kvfs"
server_key: "4e09b06804edd2409d2ba0e732f482ba"
server_key_iv: "16c92c3987280ce67b1f0e9726e25b08"
server_key_version: "encryptionkey@0"
I20260810 01:51:52.894198 11569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-1/wal/instance:
uuid: "9b061833a054483b82f3c6f79cce2414"
format_stamp: "Formatted at 2026-08-10 01:51:52 on dist-test-slave-kvfs"
server_key: "4e09b06804edd2409d2ba0e732f482ba"
server_key_iv: "16c92c3987280ce67b1f0e9726e25b08"
server_key_version: "encryptionkey@0"
I20260810 01:51:52.898811 11569 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 01:51:52.901911 11595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:52.903648 11569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 01:51:52.903838 11569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-1/wal
uuid: "9b061833a054483b82f3c6f79cce2414"
format_stamp: "Formatted at 2026-08-10 01:51:52 on dist-test-slave-kvfs"
server_key: "4e09b06804edd2409d2ba0e732f482ba"
server_key_iv: "16c92c3987280ce67b1f0e9726e25b08"
server_key_version: "encryptionkey@0"
I20260810 01:51:52.903990 11569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:52.921109 11569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:52.925217 11569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:52.925487 11569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:52.927418 11562 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.189" port: 42129 } attempt: 3
I20260810 01:51:52.928565 11565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:52.928123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:42129 (local address 127.0.0.1:51120)
0810 01:51:52.928340 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:52.928432 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.189:42129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260810 01:51:52.929554 11565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:52.929280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:42129 (local address 127.0.0.1:51134)
0810 01:51:52.929426 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:52.929485 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.189:42129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 01:51:52.929764 11562 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.189:42129: Network error: Client connection negotiation failed: client connection to 127.9.77.189:42129: connect: Connection refused (error 111)
I20260810 01:51:52.935294 11569 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.189:42129
I20260810 01:51:52.935618 11671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.189:42129 every 8 connection(s)
I20260810 01:51:52.936697 11569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-1/data/info.pb
I20260810 01:51:52.940538 11672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:52.945721 11672 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37443 } has no permanent_uuid. Determining permanent_uuid...
I20260810 01:51:52.946306 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 11569
I20260810 01:51:52.946475 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-1/wal/instance
I20260810 01:51:52.946832 9526 external_mini_cluster.cc:1428] Setting key 64239a422ec7f86ab7018acd18dea890
WARNING: no policy specified for kudu/127.9.77.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.188@KRBTEST.COM" created.
Aug 10 01:51:52 dist-test-slave-kvfs krb5kdc[11441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326712, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.189@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:51:52.968078 11566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:52.948886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42640 (local address 127.9.77.190:37443)
0810 01:51:52.949095 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:52.949101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:52.950295 (+ 1194us) server_negotiation.cc:407] Connection header received
0810 01:51:52.951511 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:52.951516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:52.951594 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:52.951721 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:52.953264 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:52.953925 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:52.955298 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:52.955683 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:52.955821 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:52.962429 (+ 6608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:52.962434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:52.962437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:52.962477 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:52.964975 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:52.965701 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:52.965705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:52.965706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:52.965808 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:52.966274 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:52.966277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:52.966279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:52.966525 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:52.966616 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:52.967556 (+ 940us) server_negotiation.cc:299] Negotiation successful
0810 01:51:52.967828 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 01:51:52.969566 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:52.949421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37443 (local address 127.0.0.1:42640)
0810 01:51:52.950105 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:52.950161 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:52.951243 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:52.951942 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:52.951950 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:52.952463 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:52.953094 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:52.953111 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:52.954088 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:52.954092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:52.955153 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:52.955163 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:52.955876 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:52.955881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:52.956168 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:52.956960 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:51:52.956988 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:52.962122 (+ 5134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:52.965139 (+ 3017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:52.965147 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:52.965165 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:52.965527 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:52.965923 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:52.965929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:52.965934 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:52.966171 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:52.966653 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:52.966664 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:52.967141 (+ 477us) client_negotiation.cc:769] Sending connection context
0810 01:51:52.967461 (+ 320us) client_negotiation.cc:240] Negotiation successful
0810 01:51:52.967982 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":206,"threads_started":1}
I20260810 01:51:52.971774 11672 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.189" port: 42129 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 01:51:52 dist-test-slave-kvfs krb5kdc[11441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326712, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.189@KRBTEST.COM for kudu/127.9.77.189@KRBTEST.COM
I20260810 01:51:52.982563 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:52.972541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:42129 (local address 127.0.0.1:51150)
0810 01:51:52.972686 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:52.972699 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:52.972778 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:52.973612 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:52.973615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:52.973819 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:52.973952 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:52.973962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:52.975087 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:52.975090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:52.975694 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:52.975703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:52.976449 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:52.976452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:52.976541 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:52.977100 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:51:52.977112 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:52.978535 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:52.980728 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:52.980731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:52.980733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:52.980945 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:52.981261 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:52.981264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:52.981266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:52.981351 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:52.981923 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:52.981926 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:52.982024 (+ 98us) client_negotiation.cc:769] Sending connection context
0810 01:51:52.982154 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 01:51:52.982322 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 01:51:52.982580 11679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:52.972731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51150 (local address 127.9.77.189:42129)
0810 01:51:52.973175 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:52.973186 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:52.973209 (+ 23us) server_negotiation.cc:407] Connection header received
0810 01:51:52.973250 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:52.973255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:52.973449 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:52.973636 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:52.974074 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:52.974953 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:52.975824 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:52.976330 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:52.976455 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:52.978670 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:52.978680 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:52.978694 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:52.978718 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:52.980598 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:52.981067 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:52.981078 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:52.981080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:52.981159 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:52.981456 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:52.981459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:52.981461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:52.981804 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:52.981981 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:52.982255 (+ 274us) server_negotiation.cc:299] Negotiation successful
0810 01:51:52.982407 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":208,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.188.keytab.
Entry for principal kudu/127.9.77.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.188.keytab.
I20260810 01:51:52.984540 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.188:38023
--webserver_interface=127.9.77.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.188.keytab
--principal=kudu/127.9.77.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:34057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:37443,127.9.77.189:42129,127.9.77.188:38023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 01:51:52.985734 11672 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.188" port: 38023 } has no permanent_uuid. Determining permanent_uuid...
I20260810 01:51:52.986876 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:52.986403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:38023 (local address 127.0.0.1:36648)
0810 01:51:52.986608 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:52.986740 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:38023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260810 01:51:52.987255 11596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.77.188:38023: connect: Connection refused (error 111)
I20260810 01:51:52.988665 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:52.988318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:38023 (local address 127.0.0.1:36654)
0810 01:51:52.988481 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:52.988558 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:38023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 01:51:52.988899 11672 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.188:38023: Network error: Client connection negotiation failed: client connection to 127.9.77.188:38023: connect: Connection refused (error 111)
I20260810 01:51:53.038295 11672 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.188" port: 38023 } attempt: 1
I20260810 01:51:53.039548 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:53.039113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:38023 (local address 127.0.0.1:36668)
0810 01:51:53.039323 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:53.039413 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:38023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 01:51:53.040441 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:53.040192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:38023 (local address 127.0.0.1:36670)
0810 01:51:53.040313 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:53.040369 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:38023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 01:51:53.040644 11672 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.188:38023: Network error: Client connection negotiation failed: client connection to 127.9.77.188:38023: connect: Connection refused (error 111)
I20260810 01:51:53.073009 11562 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.189" port: 42129 } attempt: 4
Aug 10 01:51:53 dist-test-slave-kvfs krb5kdc[11441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326712, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for kudu/127.9.77.189@KRBTEST.COM
I20260810 01:51:53.084635 11679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:53.074158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51164 (local address 127.9.77.189:42129)
0810 01:51:53.074328 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:53.074334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:53.074360 (+ 26us) server_negotiation.cc:407] Connection header received
0810 01:51:53.074553 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:53.074557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:53.074622 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:53.074717 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:53.075221 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.076176 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.077844 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.078265 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.078393 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:53.080758 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:53.080762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:53.080765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:53.080800 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:53.082750 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:53.083223 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:53.083226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:53.083227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:53.083314 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:53.083609 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:53.083613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:53.083615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:53.083840 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:53.083958 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:53.084308 (+ 350us) server_negotiation.cc:299] Negotiation successful
0810 01:51:53.084442 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 01:51:53.084635 11565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:53.073968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:42129 (local address 127.0.0.1:51164)
0810 01:51:53.074281 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:53.074298 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:53.074428 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:53.074752 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:53.074754 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:53.074970 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:53.075093 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:53.075098 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.076289 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.076292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:53.077697 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:53.077709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.078395 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.078398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:53.078481 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:53.079097 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:51:53.079109 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:53.080607 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:53.082883 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:53.082886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:53.082888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:53.083087 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:53.083414 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:53.083418 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:53.083420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:53.083510 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:53.083945 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:53.083948 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:53.084121 (+ 173us) client_negotiation.cc:769] Sending connection context
0810 01:51:53.084234 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 01:51:53.084419 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260810 01:51:53.085800 11562 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.188" port: 38023 } has no permanent_uuid. Determining permanent_uuid...
I20260810 01:51:53.086723 11565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:53.086458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:38023 (local address 127.0.0.1:36680)
0810 01:51:53.086588 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:53.086654 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:38023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 01:51:53.087525 11565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:53.087302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:38023 (local address 127.0.0.1:36690)
0810 01:51:53.087407 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:53.087468 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:38023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 01:51:53.087749 11562 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.188:38023: Network error: Client connection negotiation failed: client connection to 127.9.77.188:38023: connect: Connection refused (error 111)
I20260810 01:51:53.108872 11672 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.188" port: 38023 } attempt: 2
I20260810 01:51:53.110057 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:53.109628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:38023 (local address 127.0.0.1:36702)
0810 01:51:53.109843 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:53.109935 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:38023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 01:51:53.110973 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:53.110704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:38023 (local address 127.0.0.1:36712)
0810 01:51:53.110833 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:53.110896 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:38023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 01:51:53.111124 11672 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.188:38023: Network error: Client connection negotiation failed: client connection to 127.9.77.188:38023: connect: Connection refused (error 111)
W20260810 01:51:53.125526 11680 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 01:51:53.125866 11680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:53.125898 11680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:53.125939 11680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:53.130120 11680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:53.130187 11680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:53.130208 11680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:53.130224 11680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:53.130239 11680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:53.135939 11680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:34057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:37443,127.9.77.189:42129,127.9.77.188:38023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.188.keytab
--principal=kudu/127.9.77.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.188:38023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.77.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:53.137385 11680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:53.138736 11680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 01:51:53.147004 11562 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.188" port: 38023 } attempt: 1
I20260810 01:51:53.148334 11565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:53.147837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:38023 (local address 127.0.0.1:36714)
0810 01:51:53.148085 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:53.148178 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:38023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260810 01:51:53.148904 11693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:53.148904 11692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:53.149160 11695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:53.149480 11680 server_base.cc:1034] running on GCE node
I20260810 01:51:53.149528 11565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:53.149216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:38023 (local address 127.0.0.1:36718)
0810 01:51:53.149365 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:53.149448 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:38023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 01:51:53.149693 11562 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.188:38023: Network error: Client connection negotiation failed: client connection to 127.9.77.188:38023: connect: Connection refused (error 111)
I20260810 01:51:53.150349 11680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:53.151757 11680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:53.152956 11680 hybrid_clock.cc:648] HybridClock initialized: now 1786326713152942 us; error 71 us; skew 500 ppm
Aug 10 01:51:53 dist-test-slave-kvfs krb5kdc[11441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326713, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:53.156488 11680 init.cc:377] Logged in from keytab as kudu/127.9.77.188@KRBTEST.COM (short username kudu)
I20260810 01:51:53.157680 11680 webserver.cc:460] Webserver started at http://127.9.77.188:43243/ using document root <none> and password file <none>
I20260810 01:51:53.158416 11680 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:53.158516 11680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:53.158800 11680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:53.161482 11680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-2/data/instance:
uuid: "90d499c715694504a8eedd4c61820f8a"
format_stamp: "Formatted at 2026-08-10 01:51:53 on dist-test-slave-kvfs"
server_key: "81f3314563111947b1452e8dbf163054"
server_key_iv: "ce924a4cf232dcf239bedc6b38fe46f9"
server_key_version: "encryptionkey@0"
I20260810 01:51:53.162146 11680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-2/wal/instance:
uuid: "90d499c715694504a8eedd4c61820f8a"
format_stamp: "Formatted at 2026-08-10 01:51:53 on dist-test-slave-kvfs"
server_key: "81f3314563111947b1452e8dbf163054"
server_key_iv: "ce924a4cf232dcf239bedc6b38fe46f9"
server_key_version: "encryptionkey@0"
I20260810 01:51:53.166970 11680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 01:51:53.170142 11705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:53.171694 11680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 01:51:53.171861 11680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-2/wal
uuid: "90d499c715694504a8eedd4c61820f8a"
format_stamp: "Formatted at 2026-08-10 01:51:53 on dist-test-slave-kvfs"
server_key: "81f3314563111947b1452e8dbf163054"
server_key_iv: "ce924a4cf232dcf239bedc6b38fe46f9"
server_key_version: "encryptionkey@0"
I20260810 01:51:53.172021 11680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:53.189345 11680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:53.193099 11680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:53.193353 11680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:53.202045 11680 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.188:38023
I20260810 01:51:53.202302 11672 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.188" port: 38023 } attempt: 3
I20260810 01:51:53.202421 11779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.188:38023 every 8 connection(s)
I20260810 01:51:53.203563 11680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-2/data/info.pb
I20260810 01:51:53.205605 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 11680
I20260810 01:51:53.205783 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786326392877349-9526-0/minicluster-data/master-2/wal/instance
I20260810 01:51:53.206177 9526 external_mini_cluster.cc:1428] Setting key abd91b6f493b336d9b6f04a7953c1a7e
I20260810 01:51:53.208102 11781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 01:51:53 dist-test-slave-kvfs krb5kdc[11441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326712, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.189@KRBTEST.COM for kudu/127.9.77.188@KRBTEST.COM
Aug 10 01:51:53 dist-test-slave-kvfs krb5kdc[11441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.188@KRBTEST.COM
I20260810 01:51:53.214355 11781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37443 } has no permanent_uuid. Determining permanent_uuid...
I20260810 01:51:53.216876 11562 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.188" port: 38023 } attempt: 2
I20260810 01:51:53.219683 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:53.203070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:38023 (local address 127.0.0.1:36732)
0810 01:51:53.203283 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:53.203304 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:53.203445 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:53.206318 (+ 2873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:53.206322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:53.206583 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:53.206752 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:53.206762 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.208979 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.208986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:53.209896 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:53.209911 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.211002 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.211006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:53.211118 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:53.211824 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:51:53.211863 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:53.213843 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:53.216767 (+ 2924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:53.216774 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:53.216778 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:53.217210 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:53.217662 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:53.217669 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:53.217674 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:53.217788 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:53.218892 (+ 1104us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:53.218895 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:53.219048 (+ 153us) client_negotiation.cc:769] Sending connection context
0810 01:51:53.219178 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 01:51:53.219455 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260810 01:51:53.221467 11785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:53.207389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36734 (local address 127.9.77.188:38023)
0810 01:51:53.207719 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:53.207729 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:53.207745 (+ 16us) server_negotiation.cc:407] Connection header received
0810 01:51:53.207785 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:53.207788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:53.207843 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:53.207950 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:53.208977 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.209744 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.211421 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.211774 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.211897 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:53.215101 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:53.215104 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:53.215126 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:53.215163 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:53.216989 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:53.217554 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:53.217558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:53.217561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:53.217646 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:53.217948 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:53.217951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:53.217952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:53.218743 (+ 791us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:53.218952 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:53.219871 (+ 919us) server_negotiation.cc:299] Negotiation successful
0810 01:51:53.220236 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":306,"server-negotiator.queue_time_us":256,"spinlock_wait_cycles":5632,"thread_start_us":147,"threads_started":1}
I20260810 01:51:53.221467 11783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:53.203980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36732 (local address 127.9.77.188:38023)
0810 01:51:53.204565 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:53.204583 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:53.204629 (+ 46us) server_negotiation.cc:407] Connection header received
0810 01:51:53.205424 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:53.205443 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:53.205955 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:53.206373 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:53.206956 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.208766 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.210036 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.210842 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.211032 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:53.214099 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:53.214106 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:53.214124 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:53.214155 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:53.216616 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:53.217351 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:53.217358 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:53.217365 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:53.217540 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:53.217927 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:53.217932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:53.217935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:53.218743 (+ 808us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:53.218887 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:53.219372 (+ 485us) server_negotiation.cc:299] Negotiation successful
0810 01:51:53.220160 (+ 788us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":419,"spinlock_wait_cycles":4608,"thread_start_us":203,"threads_started":1}
Aug 10 01:51:53 dist-test-slave-kvfs krb5kdc[11441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326713, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.188@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:51:53.227808 11672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b061833a054483b82f3c6f79cce2414: Bootstrap starting.
Aug 10 01:51:53 dist-test-slave-kvfs krb5kdc[11441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:51:53.231304 11566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:53.218372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42646 (local address 127.9.77.190:37443)
0810 01:51:53.218519 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:53.218524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:53.218889 (+ 365us) server_negotiation.cc:407] Connection header received
0810 01:51:53.220265 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:53.220270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:53.220323 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:53.220379 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:53.221406 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.222116 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.223465 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.223774 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.223866 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:53.227099 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:53.227104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:53.227107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:53.227138 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:53.229076 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:53.229694 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:53.229699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:53.229701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:53.229796 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:53.230103 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:53.230107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:53.230109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:53.230315 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:53.230550 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:53.230955 (+ 405us) server_negotiation.cc:299] Negotiation successful
0810 01:51:53.231104 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 01:51:53.231674 11786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:53.217263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37443 (local address 127.0.0.1:42646)
0810 01:51:53.218726 (+ 1463us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:53.218776 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:53.220131 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:53.220546 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:53.220554 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:53.221098 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:53.221266 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:53.221274 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.222236 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.222239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:53.223333 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:53.223340 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.224051 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.224054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:53.224175 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:53.225154 (+ 979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:51:53.225169 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:53.226951 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:53.229191 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:53.229202 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:53.229210 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:53.229582 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:53.229896 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:53.229900 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:53.229905 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:53.230004 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:53.230536 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:53.230544 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:53.230699 (+ 155us) client_negotiation.cc:769] Sending connection context
0810 01:51:53.230889 (+ 190us) client_negotiation.cc:240] Negotiation successful
0810 01:51:53.231127 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1271,"thread_start_us":174,"threads_started":1}
I20260810 01:51:53.231756 11672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b061833a054483b82f3c6f79cce2414: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:53.233246 11781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.189" port: 42129 } has no permanent_uuid. Determining permanent_uuid...
I20260810 01:51:53.233044 11672 log.cc:824] T 00000000000000000000000000000000 P 9b061833a054483b82f3c6f79cce2414: Log is configured to *not* fsync() on all Append() calls
Aug 10 01:51:53 dist-test-slave-kvfs krb5kdc[11441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326712, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for kudu/127.9.77.188@KRBTEST.COM
I20260810 01:51:53.237692 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:53.224794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42654 (local address 127.9.77.190:37443)
0810 01:51:53.225171 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:53.225185 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:53.225195 (+ 10us) server_negotiation.cc:407] Connection header received
0810 01:51:53.225224 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:53.225226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:53.225260 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:53.225365 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:53.226154 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.226958 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.228378 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.228798 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.228927 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:53.232346 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:53.232351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:53.232354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:53.232380 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:53.235106 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:53.235606 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:53.235610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:53.235612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:53.235710 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:53.236062 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:53.236066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:53.236067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:53.236306 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:53.236581 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:53.237302 (+ 721us) server_negotiation.cc:299] Negotiation successful
0810 01:51:53.237462 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":120,"threads_started":1}
I20260810 01:51:53.237929 11672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b061833a054483b82f3c6f79cce2414: No bootstrap required, opened a new log
I20260810 01:51:53.239763 11565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:53.217442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:38023 (local address 127.0.0.1:36750)
0810 01:51:53.217805 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:53.217824 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:53.217929 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:53.219407 (+ 1478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:53.219411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:53.219621 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:53.219782 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:53.219794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.221044 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.221049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:53.221960 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:53.221970 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.223482 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.223486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:53.223580 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:53.233196 (+ 9616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:51:53.233215 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:53.235102 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:53.237339 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:53.237342 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:53.237345 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:53.237607 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:53.238263 (+ 656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:53.238266 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:53.238268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:53.238371 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:53.239221 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:53.239224 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:53.239336 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 01:51:53.239422 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 01:51:53.239583 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246}
Aug 10 01:51:53 dist-test-slave-kvfs krb5kdc[11441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326713, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.188@KRBTEST.COM for kudu/127.9.77.189@KRBTEST.COM
I20260810 01:51:53.241277 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:53.218078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36750 (local address 127.9.77.188:38023)
0810 01:51:53.219096 (+ 1018us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:53.219106 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:53.219120 (+ 14us) server_negotiation.cc:407] Connection header received
0810 01:51:53.219161 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:53.219167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:53.219221 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:53.219919 (+ 698us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:53.220057 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.220904 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.222101 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.223330 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.223877 (+ 547us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:53.235257 (+ 11380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:53.235261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:53.235265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:53.235290 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:53.237201 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:53.237742 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:53.237745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:53.237747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:53.238148 (+ 401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:53.238595 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:53.238598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:53.238600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:53.238842 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:53.240847 (+ 2005us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:53.240939 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 01:51:53.241079 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":947,"thread_start_us":133,"threads_started":1}
I20260810 01:51:53.243799 11672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b061833a054483b82f3c6f79cce2414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d9d7a3615749b29232b0c070524599" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37443 } } peers { permanent_uuid: "9b061833a054483b82f3c6f79cce2414" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 42129 } } peers { permanent_uuid: "90d499c715694504a8eedd4c61820f8a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 38023 } }
I20260810 01:51:53.244136 11672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b061833a054483b82f3c6f79cce2414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b061833a054483b82f3c6f79cce2414, State: Initialized, Role: FOLLOWER
I20260810 01:51:53.244575 11672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b061833a054483b82f3c6f79cce2414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d9d7a3615749b29232b0c070524599" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37443 } } peers { permanent_uuid: "9b061833a054483b82f3c6f79cce2414" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 42129 } } peers { permanent_uuid: "90d499c715694504a8eedd4c61820f8a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 38023 } }
I20260810 01:51:53.245484 11786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:53.233946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:42129 (local address 127.0.0.1:51180)
0810 01:51:53.234067 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:53.234084 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:53.234166 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:53.235192 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:53.235196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:53.235410 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:53.235560 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:53.235573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.236746 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.236750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:53.237549 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:53.237559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.238547 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.238551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:53.238652 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:53.239482 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:51:53.239501 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:53.241062 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:53.243432 (+ 2370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:53.243437 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:53.243441 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:53.243721 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:53.244090 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:53.244096 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:53.244098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:53.244203 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:53.244823 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:53.244829 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:53.244955 (+ 126us) client_negotiation.cc:769] Sending connection context
0810 01:51:53.245075 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 01:51:53.245252 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 01:51:53.245498 11562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84d9d7a3615749b29232b0c070524599: Bootstrap starting.
I20260810 01:51:53.246088 11679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:53.234159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51180 (local address 127.9.77.189:42129)
0810 01:51:53.234798 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:53.234803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:53.234819 (+ 16us) server_negotiation.cc:407] Connection header received
0810 01:51:53.234874 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:53.234880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:53.234935 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:53.235036 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:53.235696 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.236597 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.238096 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.238415 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.238552 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:53.241254 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:53.241260 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:53.241265 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:53.241295 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:53.243278 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:53.243849 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:53.243852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:53.243854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:53.243955 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:53.244336 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:53.244341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:53.244345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:53.244668 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:53.244770 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:53.245716 (+ 946us) server_negotiation.cc:299] Negotiation successful
0810 01:51:53.245876 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538}
I20260810 01:51:53.246377 11672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b061833a054483b82f3c6f79cce2414 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:53.248178 11781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.188" port: 38023 } has no permanent_uuid. Determining permanent_uuid...
I20260810 01:51:53.249857 11562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84d9d7a3615749b29232b0c070524599: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:53.250950 11562 log.cc:824] T 00000000000000000000000000000000 P 84d9d7a3615749b29232b0c070524599: Log is configured to *not* fsync() on all Append() calls
W20260810 01:51:53.253362 11814 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9b061833a054483b82f3c6f79cce2414: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 01:51:53.253446 11814 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 01:51:53.253703 11792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b061833a054483b82f3c6f79cce2414 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d9d7a3615749b29232b0c070524599" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37443 } } peers { permanent_uuid: "9b061833a054483b82f3c6f79cce2414" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 42129 } } peers { permanent_uuid: "90d499c715694504a8eedd4c61820f8a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 38023 } } }
I20260810 01:51:53.253839 11792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b061833a054483b82f3c6f79cce2414 [sys.catalog]: This master's current role is: FOLLOWER
Aug 10 01:51:53 dist-test-slave-kvfs krb5kdc[11441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326713, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.188@KRBTEST.COM for kudu/127.9.77.188@KRBTEST.COM
I20260810 01:51:53.252292 11569 master_runner.cc:186] Error getting master registration for 127.9.77.190:37443: OK, instance_id { permanent_uuid: "84d9d7a3615749b29232b0c070524599" instance_seqno: 1786326712669966 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 01:51:53.257164 11569 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 01:51:53.258486 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:53.249089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36758 (local address 127.9.77.188:38023)
0810 01:51:53.249224 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:53.249231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:53.249247 (+ 16us) server_negotiation.cc:407] Connection header received
0810 01:51:53.249287 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:53.249289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:53.249338 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:53.249431 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:53.249917 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.250720 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.251665 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.251988 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.252093 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:53.254735 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:53.254739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:53.254741 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:53.254762 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:53.256585 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:53.257037 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:53.257041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:53.257043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:53.257123 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:53.257589 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:53.257591 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:53.257592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:53.257779 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:53.257913 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:53.258171 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 01:51:53.258344 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":54}
I20260810 01:51:53.258486 11786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:53.248830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:38023 (local address 127.0.0.1:36758)
0810 01:51:53.249090 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:53.249105 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:53.249184 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:53.249468 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:53.249472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:53.249667 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:53.249787 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:53.249798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.250843 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.250846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:53.251551 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:53.251561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.252235 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.252241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:53.252326 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:53.252910 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:51:53.252929 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:53.254595 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:53.256692 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:53.256695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:53.256698 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:53.256935 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:53.257221 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:53.257224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:53.257227 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:53.257313 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:53.257903 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:53.257906 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:53.258002 (+ 96us) client_negotiation.cc:769] Sending connection context
0810 01:51:53.258112 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 01:51:53.258274 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
Aug 10 01:51:53 dist-test-slave-kvfs krb5kdc[11441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.189@KRBTEST.COM
I20260810 01:51:53.262706 11781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90d499c715694504a8eedd4c61820f8a: Bootstrap starting.
I20260810 01:51:53.262765 11562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84d9d7a3615749b29232b0c070524599: No bootstrap required, opened a new log
I20260810 01:51:53.265172 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:53.239348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51182 (local address 127.9.77.189:42129)
0810 01:51:53.251483 (+ 12135us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:53.251493 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:53.251511 (+ 18us) server_negotiation.cc:407] Connection header received
0810 01:51:53.251550 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:53.251553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:53.251600 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:53.251696 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:53.253188 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.254556 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.256302 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.256672 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.256776 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:53.260310 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:53.260315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:53.260318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:53.260347 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:53.262337 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:53.263022 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:53.263026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:53.263028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:53.263165 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:53.263495 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:53.263498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:53.263499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:53.263743 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:53.263975 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:53.264820 (+ 845us) server_negotiation.cc:299] Negotiation successful
0810 01:51:53.264966 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2194,"thread_start_us":138,"threads_started":1}
I20260810 01:51:53.266319 11781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90d499c715694504a8eedd4c61820f8a: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:53.267398 11781 log.cc:824] T 00000000000000000000000000000000 P 90d499c715694504a8eedd4c61820f8a: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:53.268364 11562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84d9d7a3615749b29232b0c070524599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d9d7a3615749b29232b0c070524599" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37443 } } peers { permanent_uuid: "9b061833a054483b82f3c6f79cce2414" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 42129 } } peers { permanent_uuid: "90d499c715694504a8eedd4c61820f8a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 38023 } }
I20260810 01:51:53.268687 11562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84d9d7a3615749b29232b0c070524599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d9d7a3615749b29232b0c070524599, State: Initialized, Role: FOLLOWER
I20260810 01:51:53.269127 11562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84d9d7a3615749b29232b0c070524599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d9d7a3615749b29232b0c070524599" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37443 } } peers { permanent_uuid: "9b061833a054483b82f3c6f79cce2414" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 42129 } } peers { permanent_uuid: "90d499c715694504a8eedd4c61820f8a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 38023 } }
I20260810 01:51:53.269347 9526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 01:51:53.269694 11823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84d9d7a3615749b29232b0c070524599 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d9d7a3615749b29232b0c070524599" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37443 } } peers { permanent_uuid: "9b061833a054483b82f3c6f79cce2414" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 42129 } } peers { permanent_uuid: "90d499c715694504a8eedd4c61820f8a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 38023 } } }
I20260810 01:51:53.269917 11823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84d9d7a3615749b29232b0c070524599 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 01:51:53.270855 11562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84d9d7a3615749b29232b0c070524599 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:53.270992 11523 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "84d9d7a3615749b29232b0c070524599"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42654
I20260810 01:51:53.271261 11523 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84d9d7a3615749b29232b0c070524599 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 01:51:53.271411 11523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84d9d7a3615749b29232b0c070524599 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:53.273054 11523 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84d9d7a3615749b29232b0c070524599 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d9d7a3615749b29232b0c070524599" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37443 } } peers { permanent_uuid: "9b061833a054483b82f3c6f79cce2414" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 42129 } } peers { permanent_uuid: "90d499c715694504a8eedd4c61820f8a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 38023 } }
I20260810 01:51:53.274312 11523 leader_election.cc:288] T 00000000000000000000000000000000 P 84d9d7a3615749b29232b0c070524599 [CANDIDATE]: Term 1 election: Requested vote from peers 9b061833a054483b82f3c6f79cce2414 (127.9.77.189:42129), 90d499c715694504a8eedd4c61820f8a (127.9.77.188:38023)
I20260810 01:51:53.274999 11632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84d9d7a3615749b29232b0c070524599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9b061833a054483b82f3c6f79cce2414"
I20260810 01:51:53.275359 11632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b061833a054483b82f3c6f79cce2414 [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:53.277083 11632 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9b061833a054483b82f3c6f79cce2414 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84d9d7a3615749b29232b0c070524599 in term 1.
I20260810 01:51:53.277598 11485 leader_election.cc:302] T 00000000000000000000000000000000 P 84d9d7a3615749b29232b0c070524599 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84d9d7a3615749b29232b0c070524599, 9b061833a054483b82f3c6f79cce2414; no voters:
W20260810 01:51:53.277863 11842 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 84d9d7a3615749b29232b0c070524599: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 01:51:53.277945 11842 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 01:51:53.278057 11823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84d9d7a3615749b29232b0c070524599 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:53.278293 11823 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84d9d7a3615749b29232b0c070524599 [term 1 LEADER]: Becoming Leader. State: Replica: 84d9d7a3615749b29232b0c070524599, State: Running, Role: LEADER
I20260810 01:51:53.278705 11823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84d9d7a3615749b29232b0c070524599 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d9d7a3615749b29232b0c070524599" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37443 } } peers { permanent_uuid: "9b061833a054483b82f3c6f79cce2414" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 42129 } } peers { permanent_uuid: "90d499c715694504a8eedd4c61820f8a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 38023 } }
I20260810 01:51:53.281318 11851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84d9d7a3615749b29232b0c070524599 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84d9d7a3615749b29232b0c070524599. Latest consensus state: current_term: 1 leader_uuid: "84d9d7a3615749b29232b0c070524599" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d9d7a3615749b29232b0c070524599" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37443 } } peers { permanent_uuid: "9b061833a054483b82f3c6f79cce2414" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 42129 } } peers { permanent_uuid: "90d499c715694504a8eedd4c61820f8a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 38023 } } }
I20260810 01:51:53.281447 11851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84d9d7a3615749b29232b0c070524599 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:53.281848 11854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 01:51:53.282970 11854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:51:53.281584 11461 master_runner.cc:186] Error getting master registration for 127.9.77.188:38023: OK, instance_id { permanent_uuid: "90d499c715694504a8eedd4c61820f8a" instance_seqno: 1786326713199642 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 01:51:53.285838 11461 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 01:51:53.286463 11781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90d499c715694504a8eedd4c61820f8a: No bootstrap required, opened a new log
I20260810 01:51:53.291586 11632 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9b061833a054483b82f3c6f79cce2414 [term 1 FOLLOWER]: Refusing update from remote peer 84d9d7a3615749b29232b0c070524599: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:51:53.292515 11823 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 84d9d7a3615749b29232b0c070524599 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b061833a054483b82f3c6f79cce2414" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 42129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:51:53.292805 11781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90d499c715694504a8eedd4c61820f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d9d7a3615749b29232b0c070524599" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37443 } } peers { permanent_uuid: "9b061833a054483b82f3c6f79cce2414" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 42129 } } peers { permanent_uuid: "90d499c715694504a8eedd4c61820f8a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 38023 } }
I20260810 01:51:53.293159 11781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90d499c715694504a8eedd4c61820f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d499c715694504a8eedd4c61820f8a, State: Initialized, Role: FOLLOWER
I20260810 01:51:53.293574 11781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90d499c715694504a8eedd4c61820f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d9d7a3615749b29232b0c070524599" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37443 } } peers { permanent_uuid: "9b061833a054483b82f3c6f79cce2414" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 42129 } } peers { permanent_uuid: "90d499c715694504a8eedd4c61820f8a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 38023 } }
I20260810 01:51:53.295377 11781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90d499c715694504a8eedd4c61820f8a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:53.295645 11749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90d499c715694504a8eedd4c61820f8a [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:53.296113 11750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84d9d7a3615749b29232b0c070524599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "90d499c715694504a8eedd4c61820f8a"
I20260810 01:51:53.297535 11749 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 90d499c715694504a8eedd4c61820f8a [term 1 FOLLOWER]: Refusing update from remote peer 84d9d7a3615749b29232b0c070524599: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:51:53.298622 11792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b061833a054483b82f3c6f79cce2414 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84d9d7a3615749b29232b0c070524599. Latest consensus state: current_term: 1 leader_uuid: "84d9d7a3615749b29232b0c070524599" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d9d7a3615749b29232b0c070524599" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37443 } } peers { permanent_uuid: "9b061833a054483b82f3c6f79cce2414" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 42129 } } peers { permanent_uuid: "90d499c715694504a8eedd4c61820f8a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 38023 } } }
I20260810 01:51:53.298766 11792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b061833a054483b82f3c6f79cce2414 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 01:51:53.299331 11823 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 84d9d7a3615749b29232b0c070524599 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d499c715694504a8eedd4c61820f8a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:51:53.298826 11865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90d499c715694504a8eedd4c61820f8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d9d7a3615749b29232b0c070524599" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37443 } } peers { permanent_uuid: "9b061833a054483b82f3c6f79cce2414" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 42129 } } peers { permanent_uuid: "90d499c715694504a8eedd4c61820f8a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 38023 } } }
I20260810 01:51:53.299880 11865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90d499c715694504a8eedd4c61820f8a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 01:51:53.300786 11750 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 90d499c715694504a8eedd4c61820f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84d9d7a3615749b29232b0c070524599 in term 1.
W20260810 01:51:53.303507 11885 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 90d499c715694504a8eedd4c61820f8a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 01:51:53.303596 11885 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 01:51:53.304726 11865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90d499c715694504a8eedd4c61820f8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84d9d7a3615749b29232b0c070524599. Latest consensus state: current_term: 1 leader_uuid: "84d9d7a3615749b29232b0c070524599" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d9d7a3615749b29232b0c070524599" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37443 } } peers { permanent_uuid: "9b061833a054483b82f3c6f79cce2414" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 42129 } } peers { permanent_uuid: "90d499c715694504a8eedd4c61820f8a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 38023 } } }
I20260810 01:51:53.304848 11865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90d499c715694504a8eedd4c61820f8a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 01:51:53.308099 11823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84d9d7a3615749b29232b0c070524599 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "84d9d7a3615749b29232b0c070524599" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d9d7a3615749b29232b0c070524599" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37443 } } peers { permanent_uuid: "9b061833a054483b82f3c6f79cce2414" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 42129 } } peers { permanent_uuid: "90d499c715694504a8eedd4c61820f8a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 38023 } } }
I20260810 01:51:53.308259 11823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84d9d7a3615749b29232b0c070524599 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:53.308439 11856 mvcc.cc:205] Tried to move back new op lower bound from 7316794217632923648 to 7316794217593544704. Current Snapshot: MvccSnapshot[applied={T|T < 7316794217632923648}]
I20260810 01:51:53.309633 11865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90d499c715694504a8eedd4c61820f8a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "84d9d7a3615749b29232b0c070524599" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d9d7a3615749b29232b0c070524599" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37443 } } peers { permanent_uuid: "9b061833a054483b82f3c6f79cce2414" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 42129 } } peers { permanent_uuid: "90d499c715694504a8eedd4c61820f8a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 38023 } } }
I20260810 01:51:53.309844 11865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90d499c715694504a8eedd4c61820f8a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 01:51:53.312476 11887 mvcc.cc:205] Tried to move back new op lower bound from 7316794217632923648 to 7316794217593544704. Current Snapshot: MvccSnapshot[applied={T|T < 7316794217632923648}]
I20260810 01:51:53.312912 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:53.289026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51194 (local address 127.9.77.189:42129)
0810 01:51:53.289153 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:53.289158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:53.292736 (+ 3578us) server_negotiation.cc:407] Connection header received
0810 01:51:53.293470 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:53.293474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:53.293537 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:53.293594 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:53.297751 (+ 4157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.298724 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.305722 (+ 6998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.306147 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.306257 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:53.308211 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:53.308216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:53.308219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:53.308253 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:53.310509 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:53.311113 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:53.311119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:53.311123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:53.311238 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:53.311606 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:53.311609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:53.311611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:53.311913 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:53.312081 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:53.312499 (+ 418us) server_negotiation.cc:299] Negotiation successful
0810 01:51:53.312683 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 01:51:53.313038 11854 catalog_manager.cc:1347] Generated new cluster ID: df29f36066454ffda0f5e7d3e27ea7b6
I20260810 01:51:53.313110 11854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:51:53.319512 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:53.288528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42660 (local address 127.9.77.190:37443)
0810 01:51:53.288662 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:53.288667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:53.293071 (+ 4404us) server_negotiation.cc:407] Connection header received
0810 01:51:53.295005 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:53.295009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:53.295067 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:53.295119 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:53.298956 (+ 3837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.299771 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.304996 (+ 5225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.305400 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.305485 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:53.310189 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:53.310196 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:53.310202 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:53.310243 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:53.312118 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:53.312693 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:53.312697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:53.312699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:53.312788 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:53.313568 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:53.313572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:53.313573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:53.313833 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:53.313958 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:53.319092 (+ 5134us) server_negotiation.cc:299] Negotiation successful
0810 01:51:53.319297 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 01:51:53.326206 11851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84d9d7a3615749b29232b0c070524599 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "84d9d7a3615749b29232b0c070524599" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d9d7a3615749b29232b0c070524599" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37443 } } peers { permanent_uuid: "9b061833a054483b82f3c6f79cce2414" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 42129 } } peers { permanent_uuid: "90d499c715694504a8eedd4c61820f8a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 38023 } } }
I20260810 01:51:53.326421 11851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84d9d7a3615749b29232b0c070524599 [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:53.328089 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:53.292105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36774 (local address 127.9.77.188:38023)
0810 01:51:53.292217 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:53.292222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:53.308414 (+ 16192us) server_negotiation.cc:407] Connection header received
0810 01:51:53.308650 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:53.308656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:53.308720 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:53.308774 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:53.312685 (+ 3911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.313435 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.320141 (+ 6706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:53.320526 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:53.320611 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:53.323120 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:53.323129 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:53.323132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:53.323174 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:53.325407 (+ 2233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:53.326053 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:53.326057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:53.326060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:53.326168 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:53.326729 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:53.326733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:53.326735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:53.327021 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:53.327179 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:53.327660 (+ 481us) server_negotiation.cc:299] Negotiation successful
0810 01:51:53.327827 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 01:51:53.327996 11792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b061833a054483b82f3c6f79cce2414 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "84d9d7a3615749b29232b0c070524599" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d9d7a3615749b29232b0c070524599" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37443 } } peers { permanent_uuid: "9b061833a054483b82f3c6f79cce2414" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 42129 } } peers { permanent_uuid: "90d499c715694504a8eedd4c61820f8a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 38023 } } }
I20260810 01:51:53.328177 11792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b061833a054483b82f3c6f79cce2414 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 01:51:53.328977 11867 mvcc.cc:205] Tried to move back new op lower bound from 7316794217632923648 to 7316794217593544704. Current Snapshot: MvccSnapshot[applied={T|T < 7316794217632923648}]
I20260810 01:51:53.358163 11854 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:51:53.359777 11854 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:51:53.371965 11854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84d9d7a3615749b29232b0c070524599: Generated new TSK 0
I20260810 01:51:53.373373 11854 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 01:51:53.416523 11789 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:51:53.413319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42672 (local address 127.9.77.190:37443)
0810 01:51:53.413491 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:53.413497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:53.414938 (+ 1441us) server_negotiation.cc:407] Connection header received
0810 01:51:53.415217 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:53.415222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:53.415293 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:53.415398 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:53.416335 (+ 937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42672: BlockingRecv error: recv got EOF from 127.0.0.1:42672 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 01:51:53.418308 11791 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:51:53.414245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51210 (local address 127.9.77.189:42129)
0810 01:51:53.414387 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:53.414394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:53.417288 (+ 2894us) server_negotiation.cc:407] Connection header received
0810 01:51:53.417444 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:53.417448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:53.417518 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:53.417608 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:53.418153 (+ 545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51210: BlockingRecv error: recv got EOF from 127.0.0.1:51210 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260810 01:51:53.420745 11787 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:51:53.419131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36786 (local address 127.9.77.188:38023)
0810 01:51:53.419290 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:53.419296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:53.419615 (+ 319us) server_negotiation.cc:407] Connection header received
0810 01:51:53.419809 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:53.419812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:53.419873 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:53.419934 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:53.420584 (+ 650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36786: BlockingRecv error: recv got EOF from 127.0.0.1:36786 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260810 01:51:53.426574 11791 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:51:53.424778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51212 (local address 127.9.77.189:42129)
0810 01:51:53.424907 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:53.424912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:53.425400 (+ 488us) server_negotiation.cc:407] Connection header received
0810 01:51:53.425658 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:53.425662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:53.425760 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:53.425824 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:53.426439 (+ 615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51212: BlockingRecv error: recv got EOF from 127.0.0.1:51212 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 01:51:53.435091 11791 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:51:53.433166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51226 (local address 127.9.77.189:42129)
0810 01:51:53.433300 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:53.433306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:53.433930 (+ 624us) server_negotiation.cc:407] Connection header received
0810 01:51:53.434163 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:53.434167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:53.434246 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:53.434321 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:53.434956 (+ 635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51226: BlockingRecv error: recv got EOF from 127.0.0.1:51226 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 01:51:53.441824 11791 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:51:53.439892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51228 (local address 127.9.77.189:42129)
0810 01:51:53.440048 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:53.440053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:53.440685 (+ 632us) server_negotiation.cc:407] Connection header received
0810 01:51:53.440931 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:53.440934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:53.441001 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:53.441069 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:53.441672 (+ 603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51228: BlockingRecv error: recv got EOF from 127.0.0.1:51228 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 01:51:53.450402 11791 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:51:53.448495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51242 (local address 127.9.77.189:42129)
0810 01:51:53.448658 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:53.448663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:53.449286 (+ 623us) server_negotiation.cc:407] Connection header received
0810 01:51:53.449506 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:53.449510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:53.449574 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:53.449637 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:53.450264 (+ 627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51242: BlockingRecv error: recv got EOF from 127.0.0.1:51242 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260810 01:51:53.463364 11791 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:51:53.461257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51248 (local address 127.9.77.189:42129)
0810 01:51:53.461408 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:53.461413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:53.462108 (+ 695us) server_negotiation.cc:407] Connection header received
0810 01:51:53.462357 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:53.462362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:53.462430 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:53.462495 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:53.463188 (+ 693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51248: BlockingRecv error: recv got EOF from 127.0.0.1:51248 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260810 01:51:53.485118 11791 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:51:53.482758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51264 (local address 127.9.77.189:42129)
0810 01:51:53.482935 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:53.482940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:53.483714 (+ 774us) server_negotiation.cc:407] Connection header received
0810 01:51:53.483977 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:53.483982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:53.484110 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:53.484178 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:53.484946 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51264: BlockingRecv error: recv got EOF from 127.0.0.1:51264 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260810 01:51:53.522186 11791 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:51:53.520455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51278 (local address 127.9.77.189:42129)
0810 01:51:53.520623 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:53.520632 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:53.520978 (+ 346us) server_negotiation.cc:407] Connection header received
0810 01:51:53.521144 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:53.521149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:53.521252 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:53.521377 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:53.522012 (+ 635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51278: BlockingRecv error: recv got EOF from 127.0.0.1:51278 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260810 01:51:53.590327 11791 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:51:53.588939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51290 (local address 127.9.77.189:42129)
0810 01:51:53.589136 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:53.589140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:53.589267 (+ 127us) server_negotiation.cc:407] Connection header received
0810 01:51:53.589435 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:53.589438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:53.589501 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:53.589598 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:53.590172 (+ 574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51290: BlockingRecv error: recv got EOF from 127.0.0.1:51290 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260810 01:51:53.722564 11791 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:51:53.721206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51298 (local address 127.9.77.189:42129)
0810 01:51:53.721383 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:53.721387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:53.721427 (+ 40us) server_negotiation.cc:407] Connection header received
0810 01:51:53.721605 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:53.721608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:53.721673 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:53.721786 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:53.722372 (+ 586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51298: BlockingRecv error: recv got EOF from 127.0.0.1:51298 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260810 01:51:53.983268 11791 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:51:53.981713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51304 (local address 127.9.77.189:42129)
0810 01:51:53.981867 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:53.981872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:53.981976 (+ 104us) server_negotiation.cc:407] Connection header received
0810 01:51:53.982204 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:53.982208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:53.982295 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:53.982365 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:53.983075 (+ 710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51304: BlockingRecv error: recv got EOF from 127.0.0.1:51304 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 01:51:54.255419 11814 catalog_manager.cc:1259] Loaded cluster ID: df29f36066454ffda0f5e7d3e27ea7b6
I20260810 01:51:54.255560 11814 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9b061833a054483b82f3c6f79cce2414: loading cluster ID for follower catalog manager: success
I20260810 01:51:54.257687 11814 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9b061833a054483b82f3c6f79cce2414: acquiring CA information for follower catalog manager: success
I20260810 01:51:54.259256 11814 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9b061833a054483b82f3c6f79cce2414: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 01:51:54.305586 11885 catalog_manager.cc:1259] Loaded cluster ID: df29f36066454ffda0f5e7d3e27ea7b6
I20260810 01:51:54.305718 11885 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 90d499c715694504a8eedd4c61820f8a: loading cluster ID for follower catalog manager: success
I20260810 01:51:54.307722 11885 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 90d499c715694504a8eedd4c61820f8a: acquiring CA information for follower catalog manager: success
I20260810 01:51:54.309322 11885 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 90d499c715694504a8eedd4c61820f8a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 01:51:54.503412 11994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:54.498570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51308 (local address 127.9.77.189:42129)
0810 01:51:54.498929 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:54.498934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:54.499074 (+ 140us) server_negotiation.cc:407] Connection header received
0810 01:51:54.499289 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:54.499293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:54.499493 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:54.499641 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:51:54.499937 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:54.500618 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:54.501584 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:54.501838 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:54.501961 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:54.502341 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:51:54.502950 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:51:54.503110 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:54.503145 (+ 35us) server_negotiation.cc:299] Negotiation successful
0810 01:51:54.503210 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":252,"threads_started":1}
I20260810 01:51:54.510970 12002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:54.506905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36792 (local address 127.9.77.188:38023)
0810 01:51:54.507200 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:54.507204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:54.507220 (+ 16us) server_negotiation.cc:407] Connection header received
0810 01:51:54.507320 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:54.507323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:54.507484 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:54.507594 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:51:54.507823 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:54.508530 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:54.509416 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:54.509646 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:54.509729 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:54.509995 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:51:54.510517 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:51:54.510615 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:54.510664 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 01:51:54.510718 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":95,"threads_started":1}
I20260810 01:51:54.519615 11994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:54.514756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51312 (local address 127.9.77.189:42129)
0810 01:51:54.514907 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:54.514914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:54.515354 (+ 440us) server_negotiation.cc:407] Connection header received
0810 01:51:54.515611 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:54.515617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:54.515685 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:54.515766 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:51:54.516097 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:54.516897 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:54.517958 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:54.518204 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:54.518320 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:54.519033 (+ 713us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:51:54.519135 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:51:54.519206 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:54.519387 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 01:51:54.519453 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 01:51:54.519614 12002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:54.514856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36808 (local address 127.9.77.188:38023)
0810 01:51:54.514969 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:54.514973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:54.515359 (+ 386us) server_negotiation.cc:407] Connection header received
0810 01:51:54.515497 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:54.515500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:54.515560 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:54.515631 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:51:54.516053 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:54.516888 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:54.518321 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:54.518573 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:54.518667 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:54.519059 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:51:54.519147 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:51:54.519210 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:54.519383 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 01:51:54.519453 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 01:51:54.520181 12008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:54.514758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42678 (local address 127.9.77.190:37443)
0810 01:51:54.515088 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:54.515096 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:54.515402 (+ 306us) server_negotiation.cc:407] Connection header received
0810 01:51:54.515684 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:54.515687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:54.515934 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:54.516112 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:51:54.516462 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:54.517099 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:54.518314 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:54.518569 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:54.518648 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:54.519156 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:51:54.519609 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:51:54.519710 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:54.519864 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 01:51:54.519956 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":86,"threads_started":1}
I20260810 01:51:54.527040 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 11461
I20260810 01:51:54.550344 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 11569
I20260810 01:51:54.567740 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 11680
2026-08-10T01:51:54Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3577 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 01:51:54 dist-test-slave-kvfs krb5kdc[12018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 01:51:54 dist-test-slave-kvfs krb5kdc[12018](info): set up 2 sockets
Aug 10 01:51:54 dist-test-slave-kvfs krb5kdc[12018](info): commencing operation
krb5kdc: starting...
W20260810 01:51:55.931910 9526 mini_kdc.cc:121] Time spent starting KDC: real 1.334s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/krb5kdc/test-user.keytab.
Aug 10 01:51:56 dist-test-slave-kvfs krb5kdc[12018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T01:51:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T01:51:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.77.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
Entry for principal kudu/127.9.77.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab.
I20260810 01:51:56.096700 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:37035
--webserver_interface=127.9.77.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:34455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:37035,127.9.77.189:40339,127.9.77.188:37643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 01:51:56.238176 12034 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 01:51:56.238651 12034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:56.238736 12034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:56.238824 12034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:56.243377 12034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:56.243511 12034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:56.243549 12034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:56.243588 12034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:56.243625 12034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:56.249748 12034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:34455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:37035,127.9.77.189:40339,127.9.77.188:37643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.190.keytab
--principal=kudu/127.9.77.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.190:37035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.77.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:56.251308 12034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:56.252705 12034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:56.263429 12044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:56.263429 12047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:56.263460 12049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:56.263864 12034 server_base.cc:1034] running on GCE node
I20260810 01:51:56.264654 12034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:56.266227 12034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:56.267480 12034 hybrid_clock.cc:648] HybridClock initialized: now 1786326716267445 us; error 51 us; skew 500 ppm
Aug 10 01:51:56 dist-test-slave-kvfs krb5kdc[12018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326716, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:56.270855 12034 init.cc:377] Logged in from keytab as kudu/127.9.77.190@KRBTEST.COM (short username kudu)
I20260810 01:51:56.271994 12034 webserver.cc:460] Webserver started at http://127.9.77.190:38731/ using document root <none> and password file <none>
I20260810 01:51:56.272799 12034 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:56.272908 12034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:56.273200 12034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:56.275836 12034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-0/data/instance:
uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b"
format_stamp: "Formatted at 2026-08-10 01:51:56 on dist-test-slave-kvfs"
server_key: "c1b4f913e52ad80c9b0a2e38c3b619d9"
server_key_iv: "6f4bb017dd9d1a9efc86bce593076c79"
server_key_version: "encryptionkey@0"
I20260810 01:51:56.276626 12034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-0/wal/instance:
uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b"
format_stamp: "Formatted at 2026-08-10 01:51:56 on dist-test-slave-kvfs"
server_key: "c1b4f913e52ad80c9b0a2e38c3b619d9"
server_key_iv: "6f4bb017dd9d1a9efc86bce593076c79"
server_key_version: "encryptionkey@0"
I20260810 01:51:56.281463 12034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 01:51:56.284658 12056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:56.286357 12034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:51:56.286561 12034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-0/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-0/wal
uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b"
format_stamp: "Formatted at 2026-08-10 01:51:56 on dist-test-slave-kvfs"
server_key: "c1b4f913e52ad80c9b0a2e38c3b619d9"
server_key_iv: "6f4bb017dd9d1a9efc86bce593076c79"
server_key_version: "encryptionkey@0"
I20260810 01:51:56.286770 12034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:56.303949 12034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:56.308002 12034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:56.308310 12034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:56.317274 12034 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.190:37035
I20260810 01:51:56.317651 12129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.190:37035 every 8 connection(s)
I20260810 01:51:56.318584 12034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-0/data/info.pb
I20260810 01:51:56.322604 12131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:56.327507 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 12034
I20260810 01:51:56.327687 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-0/wal/instance
I20260810 01:51:56.328085 9526 external_mini_cluster.cc:1428] Setting key eb9ed339cf00f226b1200412e99c33f3
I20260810 01:51:56.327781 12131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37035 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.77.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.189@KRBTEST.COM" created.
Aug 10 01:51:56 dist-test-slave-kvfs krb5kdc[12018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326716, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:51:56.350898 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.330934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34804 (local address 127.9.77.190:37035)
0810 01:51:56.331490 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:56.331517 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:56.331890 (+ 373us) server_negotiation.cc:407] Connection header received
0810 01:51:56.333219 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:56.333224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:56.333462 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:56.333684 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:56.334752 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.336174 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.337969 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.338393 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.338550 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:56.344743 (+ 6193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:56.344756 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:56.344779 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:56.344818 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:56.347120 (+ 2302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:56.347799 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:56.347804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:56.347805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:56.347928 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:56.348352 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:56.348355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:56.348356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:56.348863 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:56.349003 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:56.349519 (+ 516us) server_negotiation.cc:299] Negotiation successful
0810 01:51:56.349796 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":364,"thread_start_us":175,"threads_started":1}
I20260810 01:51:56.350893 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.330934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37035 (local address 127.0.0.1:34804)
0810 01:51:56.331708 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.331755 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:56.332840 (+ 1085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:56.333678 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:56.333683 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:56.334223 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:56.334521 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:56.334537 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.336342 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.336349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:56.337824 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:56.337842 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.338605 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.338612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:56.338765 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:56.339514 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:51:56.339547 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:56.344531 (+ 4984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:56.347247 (+ 2716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:56.347262 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:56.347277 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:56.347670 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:56.348068 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:56.348071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:56.348073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:56.348244 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:56.348996 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:56.349007 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:56.349210 (+ 203us) client_negotiation.cc:769] Sending connection context
0810 01:51:56.349432 (+ 222us) client_negotiation.cc:240] Negotiation successful
0810 01:51:56.349883 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":63,"spinlock_wait_cycles":12160,"thread_start_us":215,"threads_started":1}
I20260810 01:51:56.354312 12131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40339 } has no permanent_uuid. Determining permanent_uuid...
I20260810 01:51:56.355261 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.354955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:40339 (local address 127.0.0.1:34640)
0810 01:51:56.355104 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.355187 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.189:40339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 01:51:56.355485 12060 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.77.189:40339: connect: Connection refused (error 111)
I20260810 01:51:56.356577 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.356338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:40339 (local address 127.0.0.1:34654)
0810 01:51:56.356459 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.356516 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.189:40339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 01:51:56.356782 12131 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.189:40339: Network error: Client connection negotiation failed: client connection to 127.9.77.189:40339: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.189.keytab.
Entry for principal kudu/127.9.77.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.189.keytab.
I20260810 01:51:56.367957 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.189:40339
--webserver_interface=127.9.77.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.189.keytab
--principal=kudu/127.9.77.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:34455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:37035,127.9.77.189:40339,127.9.77.188:37643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 01:51:56.406065 12131 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40339 } attempt: 1
I20260810 01:51:56.407372 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.406946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:40339 (local address 127.0.0.1:34658)
0810 01:51:56.407160 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.407255 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.189:40339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260810 01:51:56.408388 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.408130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:40339 (local address 127.0.0.1:34668)
0810 01:51:56.408255 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.408312 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.189:40339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 01:51:56.408543 12131 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.189:40339: Network error: Client connection negotiation failed: client connection to 127.9.77.189:40339: connect: Connection refused (error 111)
I20260810 01:51:56.476719 12131 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40339 } attempt: 2
I20260810 01:51:56.477870 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.477488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:40339 (local address 127.0.0.1:34670)
0810 01:51:56.477679 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.477762 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.189:40339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 01:51:56.478842 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.478590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:40339 (local address 127.0.0.1:34672)
0810 01:51:56.478714 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.478770 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.189:40339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 01:51:56.478974 12131 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.189:40339: Network error: Client connection negotiation failed: client connection to 127.9.77.189:40339: connect: Connection refused (error 111)
W20260810 01:51:56.509070 12140 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 01:51:56.509524 12140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:56.509603 12140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:56.509683 12140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 01:51:56.514170 12140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:56.514294 12140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:56.514333 12140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:56.514374 12140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:56.514410 12140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:56.520694 12140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:34455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:37035,127.9.77.189:40339,127.9.77.188:37643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.189.keytab
--principal=kudu/127.9.77.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.189:40339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.77.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:56.522218 12140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:56.523604 12140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:56.534446 12155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:56.534451 12153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:56.534808 12140 server_base.cc:1034] running on GCE node
W20260810 01:51:56.534451 12151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:56.535728 12140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:56.537276 12140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:56.538497 12140 hybrid_clock.cc:648] HybridClock initialized: now 1786326716538482 us; error 36 us; skew 500 ppm
Aug 10 01:51:56 dist-test-slave-kvfs krb5kdc[12018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326716, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:56.541996 12140 init.cc:377] Logged in from keytab as kudu/127.9.77.189@KRBTEST.COM (short username kudu)
I20260810 01:51:56.543143 12140 webserver.cc:460] Webserver started at http://127.9.77.189:34403/ using document root <none> and password file <none>
I20260810 01:51:56.543881 12140 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:56.543957 12140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:56.544258 12140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:56.546947 12140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-1/data/instance:
uuid: "9b7299e43e80406887528f0afe8d364f"
format_stamp: "Formatted at 2026-08-10 01:51:56 on dist-test-slave-kvfs"
server_key: "2f635d4b7a71d4e9d19ef73245def3db"
server_key_iv: "4d854c5b6875950c474fb361824ff8c4"
server_key_version: "encryptionkey@0"
I20260810 01:51:56.547739 12140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-1/wal/instance:
uuid: "9b7299e43e80406887528f0afe8d364f"
format_stamp: "Formatted at 2026-08-10 01:51:56 on dist-test-slave-kvfs"
server_key: "2f635d4b7a71d4e9d19ef73245def3db"
server_key_iv: "4d854c5b6875950c474fb361824ff8c4"
server_key_version: "encryptionkey@0"
I20260810 01:51:56.552767 12140 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260810 01:51:56.555877 12166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:56.557772 12140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 01:51:56.557964 12140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-1/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-1/wal
uuid: "9b7299e43e80406887528f0afe8d364f"
format_stamp: "Formatted at 2026-08-10 01:51:56 on dist-test-slave-kvfs"
server_key: "2f635d4b7a71d4e9d19ef73245def3db"
server_key_iv: "4d854c5b6875950c474fb361824ff8c4"
server_key_version: "encryptionkey@0"
I20260810 01:51:56.558112 12140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:56.570159 12131 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40339 } attempt: 3
I20260810 01:51:56.571401 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.570980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:40339 (local address 127.0.0.1:34678)
0810 01:51:56.571192 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.571284 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.189:40339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260810 01:51:56.572414 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.572137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:40339 (local address 127.0.0.1:34688)
0810 01:51:56.572266 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.572332 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.189:40339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 01:51:56.572574 12131 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.189:40339: Network error: Client connection negotiation failed: client connection to 127.9.77.189:40339: connect: Connection refused (error 111)
I20260810 01:51:56.575798 12140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:56.580185 12140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:56.580436 12140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:56.590139 12140 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.189:40339
I20260810 01:51:56.590615 12243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.189:40339 every 8 connection(s)
I20260810 01:51:56.591652 12140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-1/data/info.pb
I20260810 01:51:56.595685 12244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:56.598851 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 12140
I20260810 01:51:56.599032 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-1/wal/instance
I20260810 01:51:56.599395 9526 external_mini_cluster.cc:1428] Setting key 05497761505bfec3fbb4dd186ff4d9f1
I20260810 01:51:56.601361 12244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37035 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.77.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.77.188@KRBTEST.COM" created.
Aug 10 01:51:56 dist-test-slave-kvfs krb5kdc[12018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326716, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.189@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:51:56.622888 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.604559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34808 (local address 127.9.77.190:37035)
0810 01:51:56.604777 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:56.604783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:56.606030 (+ 1247us) server_negotiation.cc:407] Connection header received
0810 01:51:56.607915 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:56.607919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:56.607985 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:56.608136 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:56.610445 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.611279 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.613602 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.613912 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.614030 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:56.618078 (+ 4048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:56.618082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:56.618084 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:56.618120 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:56.620233 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:56.620926 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:56.620930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:56.620932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:56.621061 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:56.621459 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:56.621465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:56.621469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:56.621732 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:56.621819 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:56.622505 (+ 686us) server_negotiation.cc:299] Negotiation successful
0810 01:51:56.622679 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 01:51:56.624078 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.605069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37035 (local address 127.0.0.1:34808)
0810 01:51:56.605789 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.605868 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:56.607574 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:56.608434 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:56.608449 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:56.609159 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:56.610197 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:56.610228 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.611476 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.611481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:56.613400 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:56.613421 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.614140 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.614144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:56.614334 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:56.615170 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:51:56.615191 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:56.617811 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:56.620417 (+ 2606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:56.620423 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:56.620446 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:56.620805 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:56.621178 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:56.621181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:56.621182 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:56.621352 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:56.621832 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:56.621838 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:56.622190 (+ 352us) client_negotiation.cc:769] Sending connection context
0810 01:51:56.622442 (+ 252us) client_negotiation.cc:240] Negotiation successful
0810 01:51:56.622880 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":164,"threads_started":1}
I20260810 01:51:56.625957 12244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40339 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 01:51:56 dist-test-slave-kvfs krb5kdc[12018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326716, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.189@KRBTEST.COM for kudu/127.9.77.189@KRBTEST.COM
I20260810 01:51:56.637257 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.626638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:40339 (local address 127.0.0.1:34692)
0810 01:51:56.626838 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.626852 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:56.626928 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:56.627665 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:56.627667 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:56.627878 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:56.627974 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:56.627980 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.629096 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.629100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:56.629665 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:56.629673 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.630348 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.630351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:56.630462 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:56.631080 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:51:56.631101 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:56.632630 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:56.635016 (+ 2386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:56.635019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:56.635021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:56.635245 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:56.635679 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:56.635682 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:56.635683 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:56.635776 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:56.636643 (+ 867us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:56.636646 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:56.636745 (+ 99us) client_negotiation.cc:769] Sending connection context
0810 01:51:56.636845 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 01:51:56.637071 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260810 01:51:56.637439 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.626765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34692 (local address 127.9.77.189:40339)
0810 01:51:56.627235 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:56.627250 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:56.627270 (+ 20us) server_negotiation.cc:407] Connection header received
0810 01:51:56.627307 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:56.627312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:56.627528 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:56.627700 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:56.628132 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.628961 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.629784 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.630223 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.630346 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:56.632816 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:56.632831 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:56.632853 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:56.632886 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:56.634847 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:56.635401 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:56.635410 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:56.635412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:56.635559 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:56.635895 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:56.635898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:56.635900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:56.636460 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:56.636604 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:56.636994 (+ 390us) server_negotiation.cc:299] Negotiation successful
0810 01:51:56.637200 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":248,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.77.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.188.keytab.
Entry for principal kudu/127.9.77.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.188.keytab.
I20260810 01:51:56.639842 12244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 } has no permanent_uuid. Determining permanent_uuid...
I20260810 01:51:56.640839 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.640501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:37643 (local address 127.0.0.1:49418)
0810 01:51:56.640652 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.640754 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:37643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260810 01:51:56.641090 12168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.77.188:37643: connect: Connection refused (error 111)
I20260810 01:51:56.641328 9526 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski_V3Up/build/debug/bin/kudu
/tmp/dist-test-taski_V3Up/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.188:37643
--webserver_interface=127.9.77.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.188.keytab
--principal=kudu/127.9.77.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.77.148:34455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.77.190:37035,127.9.77.189:40339,127.9.77.188:37643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 01:51:56.642355 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.642104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:37643 (local address 127.0.0.1:49420)
0810 01:51:56.642227 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.642287 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:37643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 01:51:56.642544 12244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.188:37643: Network error: Client connection negotiation failed: client connection to 127.9.77.188:37643: connect: Connection refused (error 111)
I20260810 01:51:56.691934 12244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 } attempt: 1
I20260810 01:51:56.693334 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.692896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:37643 (local address 127.0.0.1:49432)
0810 01:51:56.693120 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.693203 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:37643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260810 01:51:56.694244 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.693995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:37643 (local address 127.0.0.1:49438)
0810 01:51:56.694116 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.694175 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:37643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 01:51:56.694409 12244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.188:37643: Network error: Client connection negotiation failed: client connection to 127.9.77.188:37643: connect: Connection refused (error 111)
I20260810 01:51:56.715828 12131 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40339 } attempt: 4
Aug 10 01:51:56 dist-test-slave-kvfs krb5kdc[12018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326716, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for kudu/127.9.77.189@KRBTEST.COM
I20260810 01:51:56.727042 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.716897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34704 (local address 127.9.77.189:40339)
0810 01:51:56.717092 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:56.717096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:56.717181 (+ 85us) server_negotiation.cc:407] Connection header received
0810 01:51:56.717354 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:56.717356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:56.717413 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:56.717493 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:56.718173 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.718852 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.720103 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.720386 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.720489 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:56.723225 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:56.723229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:56.723231 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:56.723261 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:56.725055 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:56.725583 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:56.725588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:56.725590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:56.725691 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:56.726017 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:56.726021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:56.726023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:56.726258 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:56.726379 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:56.726681 (+ 302us) server_negotiation.cc:299] Negotiation successful
0810 01:51:56.726841 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 01:51:56.727042 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.716771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:40339 (local address 127.0.0.1:34704)
0810 01:51:56.717070 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.717091 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:56.717240 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:56.717557 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:56.717560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:56.717852 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:56.718004 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:56.718012 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.718983 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.718987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:56.719885 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:56.719897 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.720524 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.720528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:56.720664 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:56.721328 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:51:56.721346 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:56.723046 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:56.725200 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:56.725204 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:56.725207 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:56.725467 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:56.725810 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:56.725813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:56.725815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:56.725918 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:56.726365 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:56.726368 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:56.726499 (+ 131us) client_negotiation.cc:769] Sending connection context
0810 01:51:56.726619 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 01:51:56.726816 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260810 01:51:56.728585 12131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 } has no permanent_uuid. Determining permanent_uuid...
I20260810 01:51:56.729611 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.729234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:37643 (local address 127.0.0.1:49450)
0810 01:51:56.729436 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.729510 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:37643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 01:51:56.730437 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.730194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:37643 (local address 127.0.0.1:49464)
0810 01:51:56.730306 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.730363 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:37643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 01:51:56.730581 12131 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.188:37643: Network error: Client connection negotiation failed: client connection to 127.9.77.188:37643: connect: Connection refused (error 111)
I20260810 01:51:56.762598 12244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 } attempt: 2
I20260810 01:51:56.763823 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.763424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:37643 (local address 127.0.0.1:49480)
0810 01:51:56.763625 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.763713 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:37643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260810 01:51:56.764814 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.764555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:37643 (local address 127.0.0.1:49482)
0810 01:51:56.764682 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.764743 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:37643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 01:51:56.764999 12244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.188:37643: Network error: Client connection negotiation failed: client connection to 127.9.77.188:37643: connect: Connection refused (error 111)
I20260810 01:51:56.789767 12131 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 } attempt: 1
I20260810 01:51:56.791049 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.790585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:37643 (local address 127.0.0.1:49496)
0810 01:51:56.790826 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.790920 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:37643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260810 01:51:56.790723 12255 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 01:51:56.791283 12255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 01:51:56.791345 12255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 01:51:56.791405 12255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 01:51:56.791973 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.791722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:37643 (local address 127.0.0.1:49508)
0810 01:51:56.791837 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.791897 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:37643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 01:51:56.792133 12131 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.188:37643: Network error: Client connection negotiation failed: client connection to 127.9.77.188:37643: connect: Connection refused (error 111)
W20260810 01:51:56.796224 12255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 01:51:56.796289 12255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 01:51:56.796310 12255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 01:51:56.796329 12255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 01:51:56.796344 12255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 01:51:56.802472 12255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.77.148:34455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.77.190:37035,127.9.77.189:40339,127.9.77.188:37643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/krb5kdc/kudu_127.9.77.188.keytab
--principal=kudu/127.9.77.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.77.188:37643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.77.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 00:50:46 UTC on ccycloud.quasar-rjxzeq.root.comops.site
build id 4576866
I20260810 01:51:56.804073 12255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 01:51:56.805454 12255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 01:51:56.816533 12264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:56.816542 12263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 01:51:56.816542 12266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 01:51:56.816934 12255 server_base.cc:1034] running on GCE node
I20260810 01:51:56.817881 12255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 01:51:56.819408 12255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 01:51:56.820676 12255 hybrid_clock.cc:648] HybridClock initialized: now 1786326716820661 us; error 39 us; skew 500 ppm
Aug 10 01:51:56 dist-test-slave-kvfs krb5kdc[12018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326716, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 01:51:56.824467 12255 init.cc:377] Logged in from keytab as kudu/127.9.77.188@KRBTEST.COM (short username kudu)
I20260810 01:51:56.825767 12255 webserver.cc:460] Webserver started at http://127.9.77.188:44185/ using document root <none> and password file <none>
I20260810 01:51:56.826545 12255 fs_manager.cc:359] Metadata directory not provided
I20260810 01:51:56.826658 12255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 01:51:56.826968 12255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 01:51:56.829944 12255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-2/data/instance:
uuid: "82defaaede22440e905144074f301b7a"
format_stamp: "Formatted at 2026-08-10 01:51:56 on dist-test-slave-kvfs"
server_key: "c5c543a463fddfb041686606aa0d2fac"
server_key_iv: "74683e29fac0cfdfa70b63536de963a2"
server_key_version: "encryptionkey@0"
I20260810 01:51:56.830745 12255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-2/wal/instance:
uuid: "82defaaede22440e905144074f301b7a"
format_stamp: "Formatted at 2026-08-10 01:51:56 on dist-test-slave-kvfs"
server_key: "c5c543a463fddfb041686606aa0d2fac"
server_key_iv: "74683e29fac0cfdfa70b63536de963a2"
server_key_version: "encryptionkey@0"
I20260810 01:51:56.836189 12255 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 01:51:56.839617 12282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 01:51:56.841576 12255 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 01:51:56.841837 12255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-2/data,/tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-2/wal
uuid: "82defaaede22440e905144074f301b7a"
format_stamp: "Formatted at 2026-08-10 01:51:56 on dist-test-slave-kvfs"
server_key: "c5c543a463fddfb041686606aa0d2fac"
server_key_iv: "74683e29fac0cfdfa70b63536de963a2"
server_key_version: "encryptionkey@0"
I20260810 01:51:56.842052 12255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 01:51:56.856235 12244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 } attempt: 3
I20260810 01:51:56.857537 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.857081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:37643 (local address 127.0.0.1:49522)
0810 01:51:56.857324 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.857423 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:37643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 01:51:56.858493 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.858182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:37643 (local address 127.0.0.1:49530)
0810 01:51:56.858366 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.858416 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:37643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260810 01:51:56.858637 12244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.188:37643: Network error: Client connection negotiation failed: client connection to 127.9.77.188:37643: connect: Connection refused (error 111)
I20260810 01:51:56.859297 12131 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 } attempt: 2
I20260810 01:51:56.860222 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.859887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:37643 (local address 127.0.0.1:49534)
0810 01:51:56.860068 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.860130 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:37643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 01:51:56.860978 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.860768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:37643 (local address 127.0.0.1:49536)
0810 01:51:56.860876 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.860927 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.77.188:37643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 01:51:56.861147 12131 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.77.188:37643: Network error: Client connection negotiation failed: client connection to 127.9.77.188:37643: connect: Connection refused (error 111)
I20260810 01:51:56.871155 12255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 01:51:56.875396 12255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 01:51:56.875651 12255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 01:51:56.884361 12255 rpc_server.cc:307] RPC server started. Bound to: 127.9.77.188:37643
I20260810 01:51:56.884728 12362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.77.188:37643 every 8 connection(s)
I20260810 01:51:56.885640 12255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-2/data/info.pb
I20260810 01:51:56.889477 12364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 01:51:56.893118 9526 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski_V3Up/build/debug/bin/kudu as pid 12255
I20260810 01:51:56.893319 9526 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski_V3Up/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786326392877349-9526-0/minicluster-data/master-2/wal/instance
I20260810 01:51:56.893692 9526 external_mini_cluster.cc:1428] Setting key efef698e49d7f59a6b424c2c80270586
I20260810 01:51:56.894994 12364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37035 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 01:51:56 dist-test-slave-kvfs krb5kdc[12018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326716, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.188@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
Aug 10 01:51:56 dist-test-slave-kvfs krb5kdc[12018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.188@KRBTEST.COM
I20260810 01:51:56.910522 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.897426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34812 (local address 127.9.77.190:37035)
0810 01:51:56.897592 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:56.897599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:56.897941 (+ 342us) server_negotiation.cc:407] Connection header received
0810 01:51:56.898295 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:56.898299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:56.898375 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:56.898478 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:56.899657 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.900593 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.902097 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.902494 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.902614 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:56.906048 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:56.906055 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:56.906061 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:56.906100 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:56.907826 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:56.908448 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:56.908450 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:56.908452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:56.908540 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:56.908961 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:56.908965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:56.908966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:56.909257 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:56.909348 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:56.910128 (+ 780us) server_negotiation.cc:299] Negotiation successful
0810 01:51:56.910289 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 01:51:56.911782 12367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.895481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49544 (local address 127.9.77.188:37643)
0810 01:51:56.896113 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:56.896135 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:56.896185 (+ 50us) server_negotiation.cc:407] Connection header received
0810 01:51:56.897125 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:56.897157 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:56.897722 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:56.898160 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:56.899246 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.900894 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.902097 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.902762 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.902859 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:56.906520 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:56.906528 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:56.906547 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:56.906575 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:56.908710 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:56.909238 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:56.909243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:56.909244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:56.909350 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:56.909627 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:56.909630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:56.909631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:56.909933 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:56.910166 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:56.910408 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 01:51:56.910543 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":360,"thread_start_us":175,"threads_started":1}
I20260810 01:51:56.911782 12368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.897336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.190:37035 (local address 127.0.0.1:34812)
0810 01:51:56.897785 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.897833 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:56.898051 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:56.898573 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:56.898579 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:56.899122 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:56.899499 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:56.899513 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.900711 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.900714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:56.901918 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:56.901930 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.902667 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.902670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:56.902797 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:56.903774 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:51:56.903801 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:56.905834 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:56.907943 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:56.907952 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:56.907965 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:56.908344 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:56.908639 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:56.908643 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:56.908646 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:56.908845 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:56.909361 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:56.909372 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:56.909781 (+ 409us) client_negotiation.cc:769] Sending connection context
0810 01:51:56.910027 (+ 246us) client_negotiation.cc:240] Negotiation successful
0810 01:51:56.910456 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":4,"thread_start_us":105,"threads_started":1}
I20260810 01:51:56.914230 12364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40339 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 01:51:56 dist-test-slave-kvfs krb5kdc[12018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326716, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.188@KRBTEST.COM for kudu/127.9.77.189@KRBTEST.COM
Aug 10 01:51:56 dist-test-slave-kvfs krb5kdc[12018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.190@KRBTEST.COM
I20260810 01:51:56.925175 12368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.914884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.189:40339 (local address 127.0.0.1:34708)
0810 01:51:56.915119 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.915136 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:56.915224 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:56.915529 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:56.915532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:56.915755 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:56.915874 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:56.915879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.916828 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.916835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:56.917452 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:56.917460 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.917990 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.917994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:56.918119 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:56.918686 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:51:56.918698 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:56.920449 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:56.922944 (+ 2495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:56.922947 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:56.922949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:56.923180 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:56.923580 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:56.923584 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:56.923586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:56.923723 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:56.924460 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:56.924466 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:56.924595 (+ 129us) client_negotiation.cc:769] Sending connection context
0810 01:51:56.924686 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 01:51:56.924861 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260810 01:51:56.925172 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.915050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34708 (local address 127.9.77.189:40339)
0810 01:51:56.915219 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:56.915225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:56.915243 (+ 18us) server_negotiation.cc:407] Connection header received
0810 01:51:56.915341 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:56.915344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:56.915409 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:56.915518 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:56.916011 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.916681 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.917589 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.917861 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.917990 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:56.920612 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:56.920617 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:56.920620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:56.920656 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:56.922781 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:56.923318 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:56.923323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:56.923326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:56.923429 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:56.924014 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:56.924019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:56.924021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:56.924350 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:56.924434 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:56.924791 (+ 357us) server_negotiation.cc:299] Negotiation successful
0810 01:51:56.924960 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 01:51:56.925980 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.915215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34814 (local address 127.9.77.190:37035)
0810 01:51:56.915363 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:56.915366 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:56.915377 (+ 11us) server_negotiation.cc:407] Connection header received
0810 01:51:56.915410 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:56.915412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:56.915450 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:56.915548 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:56.916301 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.916856 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.917732 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.917981 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.918095 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:56.921259 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:56.921265 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:56.921273 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:56.921303 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:56.923416 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:56.924062 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:56.924066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:56.924068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:56.924156 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:56.924505 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:56.924508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:56.924509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:56.924756 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:56.925112 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:56.925725 (+ 613us) server_negotiation.cc:299] Negotiation successful
0810 01:51:56.925851 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 01:51:56.926411 12364 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 01:51:56 dist-test-slave-kvfs krb5kdc[12018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326716, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.188@KRBTEST.COM for kudu/127.9.77.188@KRBTEST.COM
Aug 10 01:51:56 dist-test-slave-kvfs krb5kdc[12018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.77.189@KRBTEST.COM
I20260810 01:51:56.935482 12367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.927200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49558 (local address 127.9.77.188:37643)
0810 01:51:56.927331 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:56.927336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:56.927352 (+ 16us) server_negotiation.cc:407] Connection header received
0810 01:51:56.927492 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:56.927495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:56.927541 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:56.927628 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:56.928061 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.928654 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.929424 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.929674 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.929788 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:56.931929 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:56.931932 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:56.931945 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:56.931966 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:56.933750 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:56.934200 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:56.934206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:56.934207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:56.934293 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:56.934567 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:56.934570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:56.934572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:56.934779 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:56.934881 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:56.935139 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 01:51:56.935307 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":53}
I20260810 01:51:56.935482 12368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.927135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:37643 (local address 127.0.0.1:49558)
0810 01:51:56.927272 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.927289 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:56.927380 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:56.927652 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:56.927654 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:56.927803 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:56.927888 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:56.927892 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.928766 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.928772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:56.929305 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:56.929312 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.929787 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.929790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:56.929858 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:56.930434 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:51:56.930449 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:56.931789 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:56.933887 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:56.933892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:56.933894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:56.934100 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:56.934387 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:56.934391 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:56.934393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:56.934473 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:56.934880 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:56.934883 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:56.934975 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 01:51:56.935081 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 01:51:56.935245 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 01:51:56.937814 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.928106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34710 (local address 127.9.77.189:40339)
0810 01:51:56.928270 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:56.928274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:56.928287 (+ 13us) server_negotiation.cc:407] Connection header received
0810 01:51:56.928328 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:56.928330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:56.928369 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:56.928458 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:56.929092 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.929598 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.930578 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.930837 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.930951 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:56.934120 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:56.934124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:56.934127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:56.934163 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:56.935922 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:56.936426 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:56.936431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:56.936432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:56.936536 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:56.936872 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:56.936874 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:56.936875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:56.937077 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:56.937293 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:56.937477 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 01:51:56.937609 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 01:51:56.940696 12364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82defaaede22440e905144074f301b7a: Bootstrap starting.
I20260810 01:51:56.942188 9526 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 01:51:56.944378 12364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82defaaede22440e905144074f301b7a: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:56.945575 12364 log.cc:824] T 00000000000000000000000000000000 P 82defaaede22440e905144074f301b7a: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:56.943921 12093 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34814
I20260810 01:51:56.948843 12364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82defaaede22440e905144074f301b7a: No bootstrap required, opened a new log
I20260810 01:51:56.954033 12364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82defaaede22440e905144074f301b7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37035 } } peers { permanent_uuid: "9b7299e43e80406887528f0afe8d364f" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40339 } } peers { permanent_uuid: "82defaaede22440e905144074f301b7a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 } }
I20260810 01:51:56.954419 12364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82defaaede22440e905144074f301b7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82defaaede22440e905144074f301b7a, State: Initialized, Role: FOLLOWER
I20260810 01:51:56.954862 12364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82defaaede22440e905144074f301b7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37035 } } peers { permanent_uuid: "9b7299e43e80406887528f0afe8d364f" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40339 } } peers { permanent_uuid: "82defaaede22440e905144074f301b7a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 } }
I20260810 01:51:56.955554 12370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82defaaede22440e905144074f301b7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37035 } } peers { permanent_uuid: "9b7299e43e80406887528f0afe8d364f" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40339 } } peers { permanent_uuid: "82defaaede22440e905144074f301b7a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 } } }
I20260810 01:51:56.955739 12370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82defaaede22440e905144074f301b7a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 01:51:56.956844 12364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82defaaede22440e905144074f301b7a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:56.961318 12131 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 } attempt: 3
W20260810 01:51:56.963696 12388 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82defaaede22440e905144074f301b7a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 01:51:56.963811 12388 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 01:51:56.964139 12255 master_runner.cc:186] Error getting master registration for 127.9.77.190:37035: OK, instance_id { permanent_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" instance_seqno: 1786326716314750 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 01:51:56.968947 12255 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 10 01:51:56 dist-test-slave-kvfs krb5kdc[12018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326716, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.190@KRBTEST.COM for kudu/127.9.77.188@KRBTEST.COM
I20260810 01:51:56.974148 12367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.962579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49574 (local address 127.9.77.188:37643)
0810 01:51:56.962745 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:56.962750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:56.962765 (+ 15us) server_negotiation.cc:407] Connection header received
0810 01:51:56.962826 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:56.962829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:56.962884 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:56.962965 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:56.963979 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.964666 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.966031 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.966468 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.966587 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:56.969869 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:56.969873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:56.969877 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:56.969911 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:56.971932 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:56.972520 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:56.972524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:56.972526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:56.972624 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:56.972973 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:56.972976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:56.972978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:56.973324 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:56.973438 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:56.973775 (+ 337us) server_negotiation.cc:299] Negotiation successful
0810 01:51:56.973927 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 01:51:56.974164 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:56.961924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:37643 (local address 127.0.0.1:49574)
0810 01:51:56.962255 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:56.962274 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:56.962399 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:56.963344 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:56.963348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:56.963652 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:56.963836 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:56.963845 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.964839 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.964844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:56.965875 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:56.965890 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:56.966621 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:56.966627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:56.966849 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:56.967630 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:51:56.967648 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:56.969690 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:56.972087 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:56.972091 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:56.972094 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:56.972403 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:56.972734 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:56.972738 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:56.972740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:56.972866 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:56.973441 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:56.973445 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:56.973579 (+ 134us) client_negotiation.cc:769] Sending connection context
0810 01:51:56.973701 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 01:51:56.973911 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224}
I20260810 01:51:56.978756 12131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d476b2d2f7a4b29a510bc9d3fa2e43b: Bootstrap starting.
I20260810 01:51:56.981990 12131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d476b2d2f7a4b29a510bc9d3fa2e43b: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:56.982978 12131 log.cc:824] T 00000000000000000000000000000000 P 1d476b2d2f7a4b29a510bc9d3fa2e43b: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:56.985450 12131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d476b2d2f7a4b29a510bc9d3fa2e43b: No bootstrap required, opened a new log
I20260810 01:51:56.988495 12131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d476b2d2f7a4b29a510bc9d3fa2e43b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37035 } } peers { permanent_uuid: "9b7299e43e80406887528f0afe8d364f" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40339 } } peers { permanent_uuid: "82defaaede22440e905144074f301b7a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 } }
I20260810 01:51:56.988780 12131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d476b2d2f7a4b29a510bc9d3fa2e43b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d476b2d2f7a4b29a510bc9d3fa2e43b, State: Initialized, Role: FOLLOWER
I20260810 01:51:56.989182 12131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d476b2d2f7a4b29a510bc9d3fa2e43b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37035 } } peers { permanent_uuid: "9b7299e43e80406887528f0afe8d364f" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40339 } } peers { permanent_uuid: "82defaaede22440e905144074f301b7a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 } }
I20260810 01:51:56.989774 12391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d476b2d2f7a4b29a510bc9d3fa2e43b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37035 } } peers { permanent_uuid: "9b7299e43e80406887528f0afe8d364f" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40339 } } peers { permanent_uuid: "82defaaede22440e905144074f301b7a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 } } }
I20260810 01:51:56.989924 12391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d476b2d2f7a4b29a510bc9d3fa2e43b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 01:51:56.990849 12131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d476b2d2f7a4b29a510bc9d3fa2e43b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:56.990993 12093 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d476b2d2f7a4b29a510bc9d3fa2e43b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 01:51:56.991205 12093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d476b2d2f7a4b29a510bc9d3fa2e43b [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:56.992897 12093 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d476b2d2f7a4b29a510bc9d3fa2e43b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37035 } } peers { permanent_uuid: "9b7299e43e80406887528f0afe8d364f" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40339 } } peers { permanent_uuid: "82defaaede22440e905144074f301b7a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 } }
I20260810 01:51:56.995359 12093 leader_election.cc:288] T 00000000000000000000000000000000 P 1d476b2d2f7a4b29a510bc9d3fa2e43b [CANDIDATE]: Term 1 election: Requested vote from peers 9b7299e43e80406887528f0afe8d364f (127.9.77.189:40339), 82defaaede22440e905144074f301b7a (127.9.77.188:37643)
I20260810 01:51:56.996073 12316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "82defaaede22440e905144074f301b7a"
I20260810 01:51:56.996433 12316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82defaaede22440e905144074f301b7a [term 0 FOLLOWER]: Advancing to term 1
W20260810 01:51:56.998104 12402 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d476b2d2f7a4b29a510bc9d3fa2e43b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 01:51:56.998204 12402 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 01:51:56.998340 12316 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 82defaaede22440e905144074f301b7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d476b2d2f7a4b29a510bc9d3fa2e43b in term 1.
I20260810 01:51:56.999167 12058 leader_election.cc:302] T 00000000000000000000000000000000 P 1d476b2d2f7a4b29a510bc9d3fa2e43b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d476b2d2f7a4b29a510bc9d3fa2e43b, 82defaaede22440e905144074f301b7a; no voters:
I20260810 01:51:56.999781 12391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d476b2d2f7a4b29a510bc9d3fa2e43b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 01:51:57.000169 12391 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d476b2d2f7a4b29a510bc9d3fa2e43b [term 1 LEADER]: Becoming Leader. State: Replica: 1d476b2d2f7a4b29a510bc9d3fa2e43b, State: Running, Role: LEADER
I20260810 01:51:57.000597 12391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d476b2d2f7a4b29a510bc9d3fa2e43b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37035 } } peers { permanent_uuid: "9b7299e43e80406887528f0afe8d364f" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40339 } } peers { permanent_uuid: "82defaaede22440e905144074f301b7a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 } }
I20260810 01:51:57.002951 12391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d476b2d2f7a4b29a510bc9d3fa2e43b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d476b2d2f7a4b29a510bc9d3fa2e43b. Latest consensus state: current_term: 1 leader_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37035 } } peers { permanent_uuid: "9b7299e43e80406887528f0afe8d364f" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40339 } } peers { permanent_uuid: "82defaaede22440e905144074f301b7a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 } } }
I20260810 01:51:57.003084 12391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d476b2d2f7a4b29a510bc9d3fa2e43b [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:57.003345 12244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 } attempt: 4
I20260810 01:51:57.002388 12034 master_runner.cc:186] Error getting master registration for 127.9.77.189:40339: OK, instance_id { permanent_uuid: "9b7299e43e80406887528f0afe8d364f" instance_seqno: 1786326716587179 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 01:51:57.006871 12034 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 01:51:57.007563 12415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 10 01:51:57 dist-test-slave-kvfs krb5kdc[12018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786326716, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.77.189@KRBTEST.COM for kudu/127.9.77.188@KRBTEST.COM
I20260810 01:51:57.012897 12415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 01:51:57.016111 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:57.003856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.77.188:37643 (local address 127.0.0.1:49586)
0810 01:51:57.004234 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0810 01:51:57.004257 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 01:51:57.004360 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 01:51:57.005092 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 01:51:57.005096 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 01:51:57.005371 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0810 01:51:57.005518 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:57.005526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:57.006728 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:57.006745 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:57.007710 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 01:51:57.007724 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 01:51:57.008571 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 01:51:57.008579 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 01:51:57.008678 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:57.009441 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 01:51:57.009463 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 01:51:57.011473 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 01:51:57.014069 (+ 2596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:57.014073 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:57.014076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:57.014393 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:57.014772 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 01:51:57.014775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 01:51:57.014777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 01:51:57.014886 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 01:51:57.015440 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 01:51:57.015444 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 01:51:57.015567 (+ 123us) client_negotiation.cc:769] Sending connection context
0810 01:51:57.015682 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 01:51:57.015872 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262}
I20260810 01:51:57.016813 12367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:57.004542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49586 (local address 127.9.77.188:37643)
0810 01:51:57.004692 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:57.004701 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:57.004715 (+ 14us) server_negotiation.cc:407] Connection header received
0810 01:51:57.004781 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:57.004784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:57.004842 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:57.004929 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:57.005680 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:57.006567 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:57.007904 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:57.008288 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:57.008406 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:57.011671 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:57.011676 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:57.011680 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:57.011712 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:57.013925 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:57.014524 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:57.014528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:57.014530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:57.014639 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:57.015008 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:57.015012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:57.015014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:57.015278 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:57.015388 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:57.016463 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0810 01:51:57.016618 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 01:51:57.021207 12244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b7299e43e80406887528f0afe8d364f: Bootstrap starting.
I20260810 01:51:57.025058 12244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b7299e43e80406887528f0afe8d364f: Neither blocks nor log segments found. Creating new log.
I20260810 01:51:57.026391 12244 log.cc:824] T 00000000000000000000000000000000 P 9b7299e43e80406887528f0afe8d364f: Log is configured to *not* fsync() on all Append() calls
I20260810 01:51:57.027293 12316 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 82defaaede22440e905144074f301b7a [term 1 FOLLOWER]: Refusing update from remote peer 1d476b2d2f7a4b29a510bc9d3fa2e43b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:51:57.029474 12244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b7299e43e80406887528f0afe8d364f: No bootstrap required, opened a new log
I20260810 01:51:57.030530 12404 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d476b2d2f7a4b29a510bc9d3fa2e43b [LEADER]: Connected to new peer: Peer: permanent_uuid: "82defaaede22440e905144074f301b7a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:51:57.033460 12418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:57.008566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49594 (local address 127.9.77.188:37643)
0810 01:51:57.012890 (+ 4324us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:57.012899 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:57.013437 (+ 538us) server_negotiation.cc:407] Connection header received
0810 01:51:57.013478 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:57.013482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:57.013532 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:57.013617 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:57.014665 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:57.015580 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:57.021701 (+ 6121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:57.022117 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:57.022217 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:57.024180 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:57.024185 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:57.024188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:57.024218 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:57.029057 (+ 4839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:57.030192 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:57.030224 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:57.030227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:57.030389 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:57.031109 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:57.031114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:57.031116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:57.031615 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:57.031849 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:57.032509 (+ 660us) server_negotiation.cc:299] Negotiation successful
0810 01:51:57.032762 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4233,"thread_start_us":132,"threads_started":1}
I20260810 01:51:57.034780 12244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b7299e43e80406887528f0afe8d364f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37035 } } peers { permanent_uuid: "9b7299e43e80406887528f0afe8d364f" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40339 } } peers { permanent_uuid: "82defaaede22440e905144074f301b7a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 } }
I20260810 01:51:57.035082 12244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b7299e43e80406887528f0afe8d364f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b7299e43e80406887528f0afe8d364f, State: Initialized, Role: FOLLOWER
I20260810 01:51:57.035514 12244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b7299e43e80406887528f0afe8d364f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37035 } } peers { permanent_uuid: "9b7299e43e80406887528f0afe8d364f" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40339 } } peers { permanent_uuid: "82defaaede22440e905144074f301b7a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 } }
I20260810 01:51:57.035770 12370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82defaaede22440e905144074f301b7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d476b2d2f7a4b29a510bc9d3fa2e43b. Latest consensus state: current_term: 1 leader_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37035 } } peers { permanent_uuid: "9b7299e43e80406887528f0afe8d364f" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40339 } } peers { permanent_uuid: "82defaaede22440e905144074f301b7a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 } } }
I20260810 01:51:57.035924 12370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82defaaede22440e905144074f301b7a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 01:51:57.037420 12244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b7299e43e80406887528f0afe8d364f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 01:51:57.037482 12205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9b7299e43e80406887528f0afe8d364f"
I20260810 01:51:57.037719 12204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b7299e43e80406887528f0afe8d364f [term 0 FOLLOWER]: Advancing to term 1
I20260810 01:51:57.039631 12204 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9b7299e43e80406887528f0afe8d364f [term 1 FOLLOWER]: Refusing update from remote peer 1d476b2d2f7a4b29a510bc9d3fa2e43b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 01:51:57.039929 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:57.007333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34820 (local address 127.9.77.190:37035)
0810 01:51:57.007480 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:57.007485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:57.012655 (+ 5170us) server_negotiation.cc:407] Connection header received
0810 01:51:57.013404 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:57.013408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:57.013476 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:57.013533 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:57.017816 (+ 4283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:57.018604 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:57.020722 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:57.021074 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:57.021155 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:57.028346 (+ 7191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:57.028350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:57.028353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:57.028387 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:57.030359 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:57.038250 (+ 7891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:57.038254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:57.038256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:57.038403 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:57.038728 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:57.038731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:57.038733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:57.039000 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:57.039140 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:57.039585 (+ 445us) server_negotiation.cc:299] Negotiation successful
0810 01:51:57.039754 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 01:51:57.040750 12404 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d476b2d2f7a4b29a510bc9d3fa2e43b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b7299e43e80406887528f0afe8d364f" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 01:51:57.041805 12205 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9b7299e43e80406887528f0afe8d364f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d476b2d2f7a4b29a510bc9d3fa2e43b in term 1.
I20260810 01:51:57.049979 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:57.008265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34724 (local address 127.9.77.189:40339)
0810 01:51:57.008411 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:57.008416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:57.013147 (+ 4731us) server_negotiation.cc:407] Connection header received
0810 01:51:57.014108 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:57.014112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:57.014178 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:57.014240 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:57.015805 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:57.016585 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:57.020123 (+ 3538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:57.020480 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:57.020567 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:57.045474 (+ 24907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 01:51:57.045478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 01:51:57.045482 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 01:51:57.045522 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 01:51:57.047742 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:57.048483 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:57.048487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:57.048489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:57.048583 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 01:51:57.048890 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 01:51:57.048893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 01:51:57.048895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 01:51:57.049143 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 01:51:57.049263 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:57.049646 (+ 383us) server_negotiation.cc:299] Negotiation successful
0810 01:51:57.049795 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 01:51:57.055256 12404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d476b2d2f7a4b29a510bc9d3fa2e43b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37035 } } peers { permanent_uuid: "9b7299e43e80406887528f0afe8d364f" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40339 } } peers { permanent_uuid: "82defaaede22440e905144074f301b7a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 } } }
I20260810 01:51:57.055451 12404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d476b2d2f7a4b29a510bc9d3fa2e43b [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:57.056078 12421 mvcc.cc:205] Tried to move back new op lower bound from 7316794232929517568 to 7316794232838926336. Current Snapshot: MvccSnapshot[applied={T|T < 7316794232929517568}]
I20260810 01:51:57.057353 12370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82defaaede22440e905144074f301b7a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37035 } } peers { permanent_uuid: "9b7299e43e80406887528f0afe8d364f" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40339 } } peers { permanent_uuid: "82defaaede22440e905144074f301b7a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 } } }
I20260810 01:51:57.057512 12370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82defaaede22440e905144074f301b7a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 01:51:57.058743 12425 mvcc.cc:205] Tried to move back new op lower bound from 7316794232929517568 to 7316794232838926336. Current Snapshot: MvccSnapshot[applied={T|T < 7316794232929517568}]
I20260810 01:51:57.060966 12443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b7299e43e80406887528f0afe8d364f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37035 } } peers { permanent_uuid: "9b7299e43e80406887528f0afe8d364f" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40339 } } peers { permanent_uuid: "82defaaede22440e905144074f301b7a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 } } }
I20260810 01:51:57.061121 12443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b7299e43e80406887528f0afe8d364f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 01:51:57.061221 12443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b7299e43e80406887528f0afe8d364f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d476b2d2f7a4b29a510bc9d3fa2e43b. Latest consensus state: current_term: 1 leader_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37035 } } peers { permanent_uuid: "9b7299e43e80406887528f0afe8d364f" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40339 } } peers { permanent_uuid: "82defaaede22440e905144074f301b7a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 } } }
I20260810 01:51:57.061316 12443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b7299e43e80406887528f0afe8d364f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 01:51:57.064829 12391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d476b2d2f7a4b29a510bc9d3fa2e43b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37035 } } peers { permanent_uuid: "9b7299e43e80406887528f0afe8d364f" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40339 } } peers { permanent_uuid: "82defaaede22440e905144074f301b7a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 } } }
I20260810 01:51:57.064994 12391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d476b2d2f7a4b29a510bc9d3fa2e43b [sys.catalog]: This master's current role is: LEADER
I20260810 01:51:57.066704 12415 catalog_manager.cc:1347] Generated new cluster ID: 7edcaedda4174a529cad05ef69cdaa2e
I20260810 01:51:57.066797 12415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 01:51:57.067505 12443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b7299e43e80406887528f0afe8d364f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d476b2d2f7a4b29a510bc9d3fa2e43b" member_type: VOTER last_known_addr { host: "127.9.77.190" port: 37035 } } peers { permanent_uuid: "9b7299e43e80406887528f0afe8d364f" member_type: VOTER last_known_addr { host: "127.9.77.189" port: 40339 } } peers { permanent_uuid: "82defaaede22440e905144074f301b7a" member_type: VOTER last_known_addr { host: "127.9.77.188" port: 37643 } } }
I20260810 01:51:57.067641 12443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b7299e43e80406887528f0afe8d364f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 01:51:57.067950 12444 mvcc.cc:205] Tried to move back new op lower bound from 7316794232929517568 to 7316794232838926336. Current Snapshot: MvccSnapshot[applied={T|T < 7316794232929517568}]
I20260810 01:51:57.074385 12446 catalog_manager.cc:1259] Loaded cluster ID: 7edcaedda4174a529cad05ef69cdaa2e
I20260810 01:51:57.074471 12446 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9b7299e43e80406887528f0afe8d364f: loading cluster ID for follower catalog manager: success
W20260810 01:51:57.075026 12446 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 9b7299e43e80406887528f0afe8d364f: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260810 01:51:57.075131 12446 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260810 01:51:57.138814 12415 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 01:51:57.140640 12415 catalog_manager.cc:1504] Loading token signing keys...
I20260810 01:51:57.165186 12415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d476b2d2f7a4b29a510bc9d3fa2e43b: Generated new TSK 0
I20260810 01:51:57.166359 12415 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 01:51:57.215205 12135 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:51:57.211282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34824 (local address 127.9.77.190:37035)
0810 01:51:57.211468 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:57.211474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:57.213665 (+ 2191us) server_negotiation.cc:407] Connection header received
0810 01:51:57.213930 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:57.213935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:57.214024 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:57.214094 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:57.214990 (+ 896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34824: BlockingRecv error: recv got EOF from 127.0.0.1:34824 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260810 01:51:57.216113 12252 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:51:57.212259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34740 (local address 127.9.77.189:40339)
0810 01:51:57.212396 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:57.212401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:57.214865 (+ 2464us) server_negotiation.cc:407] Connection header received
0810 01:51:57.215085 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:57.215088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:57.215157 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:57.215220 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:57.215914 (+ 694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34740: BlockingRecv error: recv got EOF from 127.0.0.1:34740 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 01:51:57.217114 12418 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:51:57.213060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49608 (local address 127.9.77.188:37643)
0810 01:51:57.213223 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:57.213228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:57.215827 (+ 2599us) server_negotiation.cc:407] Connection header received
0810 01:51:57.216134 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:57.216140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:57.216220 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:57.216284 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:57.216974 (+ 690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49608: BlockingRecv error: recv got EOF from 127.0.0.1:49608 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260810 01:51:57.220557 12252 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:51:57.219299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34746 (local address 127.9.77.189:40339)
0810 01:51:57.219430 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:57.219434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:57.219604 (+ 170us) server_negotiation.cc:407] Connection header received
0810 01:51:57.219803 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:57.219806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:57.219886 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:57.219992 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:57.220436 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34746: BlockingRecv error: recv got EOF from 127.0.0.1:34746 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260810 01:51:57.224999 12252 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:51:57.223986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34752 (local address 127.9.77.189:40339)
0810 01:51:57.224219 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:57.224225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:57.224242 (+ 17us) server_negotiation.cc:407] Connection header received
0810 01:51:57.224294 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:57.224296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:57.224344 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:57.224450 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:57.224880 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34752: BlockingRecv error: recv got EOF from 127.0.0.1:34752 (error 108)
Metrics: {"server-negotiator.queue_time_us":151}
W20260810 01:51:57.230331 12252 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:51:57.229077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34756 (local address 127.9.77.189:40339)
0810 01:51:57.229205 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:57.229210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:57.229451 (+ 241us) server_negotiation.cc:407] Connection header received
0810 01:51:57.229590 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:57.229593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:57.229645 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:57.229741 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:57.230204 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34756: BlockingRecv error: recv got EOF from 127.0.0.1:34756 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260810 01:51:57.238186 12252 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:51:57.236769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34758 (local address 127.9.77.189:40339)
0810 01:51:57.236917 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:57.236924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:57.237147 (+ 223us) server_negotiation.cc:407] Connection header received
0810 01:51:57.237321 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:57.237324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:57.237390 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:57.237494 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:57.238024 (+ 530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34758: BlockingRecv error: recv got EOF from 127.0.0.1:34758 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260810 01:51:57.254233 12252 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:51:57.252609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34760 (local address 127.9.77.189:40339)
0810 01:51:57.252798 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:57.252803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:57.253109 (+ 306us) server_negotiation.cc:407] Connection header received
0810 01:51:57.253316 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:57.253320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:57.253396 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:57.253502 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:57.254082 (+ 580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34760: BlockingRecv error: recv got EOF from 127.0.0.1:34760 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260810 01:51:57.274821 12252 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:51:57.273387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34772 (local address 127.9.77.189:40339)
0810 01:51:57.273558 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:57.273563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:57.273748 (+ 185us) server_negotiation.cc:407] Connection header received
0810 01:51:57.273918 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:57.273922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:57.273995 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:57.274102 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:57.274646 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34772: BlockingRecv error: recv got EOF from 127.0.0.1:34772 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260810 01:51:57.310978 12252 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:51:57.309630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34776 (local address 127.9.77.189:40339)
0810 01:51:57.309852 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:57.309855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:57.309878 (+ 23us) server_negotiation.cc:407] Connection header received
0810 01:51:57.310096 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:57.310099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:57.310168 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:57.310266 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:57.310828 (+ 562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34776: BlockingRecv error: recv got EOF from 127.0.0.1:34776 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
W20260810 01:51:57.379411 12252 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:51:57.377995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34786 (local address 127.9.77.189:40339)
0810 01:51:57.378158 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:57.378161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:57.378227 (+ 66us) server_negotiation.cc:407] Connection header received
0810 01:51:57.378389 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:57.378392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:57.378457 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:57.378589 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:57.379190 (+ 601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34786: BlockingRecv error: recv got EOF from 127.0.0.1:34786 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260810 01:51:57.512390 12252 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:51:57.510527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34794 (local address 127.9.77.189:40339)
0810 01:51:57.510723 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:57.510728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:57.511018 (+ 290us) server_negotiation.cc:407] Connection header received
0810 01:51:57.511210 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:57.511214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:57.511323 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:57.511425 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:57.512222 (+ 797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34794: BlockingRecv error: recv got EOF from 127.0.0.1:34794 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260810 01:51:57.772287 12252 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 01:51:57.770984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34802 (local address 127.9.77.189:40339)
0810 01:51:57.771143 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:57.771147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:57.771241 (+ 94us) server_negotiation.cc:407] Connection header received
0810 01:51:57.771391 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:57.771394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:57.771455 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:57.771574 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 01:51:57.772151 (+ 577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34802: BlockingRecv error: recv got EOF from 127.0.0.1:34802 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 01:51:57.965674 12388 catalog_manager.cc:1259] Loaded cluster ID: 7edcaedda4174a529cad05ef69cdaa2e
I20260810 01:51:57.965794 12388 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 82defaaede22440e905144074f301b7a: loading cluster ID for follower catalog manager: success
I20260810 01:51:57.967813 12388 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 82defaaede22440e905144074f301b7a: acquiring CA information for follower catalog manager: success
I20260810 01:51:57.969319 12388 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 82defaaede22440e905144074f301b7a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 01:51:58.078171 12446 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9b7299e43e80406887528f0afe8d364f: acquiring CA information for follower catalog manager: success
I20260810 01:51:58.079772 12446 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9b7299e43e80406887528f0afe8d364f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 01:51:58.291703 12560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:58.287111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34810 (local address 127.9.77.189:40339)
0810 01:51:58.287416 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:58.287420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:58.287439 (+ 19us) server_negotiation.cc:407] Connection header received
0810 01:51:58.287562 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:58.287565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:58.287742 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:58.287884 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:51:58.288181 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:58.288808 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:58.289769 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:58.290020 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:58.290136 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:58.290524 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:51:58.291222 (+ 698us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:51:58.291372 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:58.291408 (+ 36us) server_negotiation.cc:299] Negotiation successful
0810 01:51:58.291477 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":91,"threads_started":1}
I20260810 01:51:58.299935 12568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:58.295632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49618 (local address 127.9.77.188:37643)
0810 01:51:58.295938 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:58.295943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:58.295975 (+ 32us) server_negotiation.cc:407] Connection header received
0810 01:51:58.296144 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:58.296146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:58.296311 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:58.296432 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:51:58.296690 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:58.297325 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:58.298169 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:58.298466 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:58.298550 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:58.298902 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:51:58.299511 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:51:58.299623 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:58.299680 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 01:51:58.299742 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":112,"threads_started":1}
I20260810 01:51:58.308372 12560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:58.304073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34826 (local address 127.9.77.189:40339)
0810 01:51:58.304229 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:58.304234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:58.304582 (+ 348us) server_negotiation.cc:407] Connection header received
0810 01:51:58.304806 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:58.304809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:58.304887 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:58.304979 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:51:58.305244 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:58.306047 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:58.307000 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:58.307306 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:58.307413 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:58.307719 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:51:58.307832 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:51:58.307903 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:58.308087 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 01:51:58.308177 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 01:51:58.308871 12568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:58.303999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49632 (local address 127.9.77.188:37643)
0810 01:51:58.304152 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:58.304158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:58.304424 (+ 266us) server_negotiation.cc:407] Connection header received
0810 01:51:58.304600 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:58.304604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:58.304664 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:58.304758 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:51:58.305339 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:58.305869 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:58.307266 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:58.307595 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:58.307713 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:58.308188 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:51:58.308286 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:51:58.308345 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:58.308647 (+ 302us) server_negotiation.cc:299] Negotiation successful
0810 01:51:58.308712 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 01:51:58.309480 12577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 01:51:58.303731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34834 (local address 127.9.77.190:37035)
0810 01:51:58.304243 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0810 01:51:58.304250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 01:51:58.304267 (+ 17us) server_negotiation.cc:407] Connection header received
0810 01:51:58.304446 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 01:51:58.304450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 01:51:58.304685 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 01:51:58.304838 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 01:51:58.305074 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:58.305982 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:58.307301 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 01:51:58.307703 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 01:51:58.307826 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 01:51:58.308302 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 01:51:58.308766 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 01:51:58.309186 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0810 01:51:58.309223 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 01:51:58.309310 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":101,"threads_started":1}
I20260810 01:51:58.313450 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 12034
I20260810 01:51:58.338002 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 12140
I20260810 01:51:58.356014 9526 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski_V3Up/build/debug/bin/kudu with pid 12255
2026-08-10T01:51:58Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3789 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (7367 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (325476 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST