Note: This is test shard 3 of 6.
[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 6 tests from RaftConsensusITest
[ RUN ] RaftConsensusITest.TestFailedOp
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260619 08:54:54.842713 392 test_util.cc:268] Using random seed: 1917783867
I20260619 08:54:54.848526 392 ts_itest-base.cc:112] Starting cluster with:
I20260619 08:54:54.848591 392 ts_itest-base.cc:113] --------------
I20260619 08:54:54.848610 392 ts_itest-base.cc:114] 3 tablet servers
I20260619 08:54:54.848675 392 ts_itest-base.cc:115] 3 replicas per TS
I20260619 08:54:54.848702 392 ts_itest-base.cc:116] --------------
2026-06-19T08:54:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:54:54Z Disabled control of system clock
I20260619 08:54:54.856457 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXnrr3A/build/release/bin/kudu
/tmp/dist-test-taskXnrr3A/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38311
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38311 with env {}
W20260619 08:54:54.932390 404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:54:54.932587 404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:54:54.932615 404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:54:54.933624 404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:54:54.933681 404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:54:54.933701 404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:54:54.933719 404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:54:54.934994 404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38311
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:54:54.935245 404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:54:54.935515 404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:54:54.940412 412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:54:54.940407 409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:54:54.940569 410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:54:54.940572 404 server_base.cc:1034] running on GCE node
I20260619 08:54:54.940879 404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:54:54.941165 404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:54:54.942306 404 hybrid_clock.cc:648] HybridClock initialized: now 1781859294942303 us; error 39 us; skew 500 ppm
I20260619 08:54:54.943519 404 webserver.cc:460] Webserver started at http://127.0.98.62:34199/ using document root <none> and password file <none>
I20260619 08:54:54.943737 404 fs_manager.cc:359] Metadata directory not provided
I20260619 08:54:54.943802 404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:54:54.943945 404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:54:54.944875 404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "571a7f3960cd4057a52a5ba90f316650"
format_stamp: "Formatted at 2026-06-19 08:54:54 on dist-test-slave-k50r"
I20260619 08:54:54.945221 404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "571a7f3960cd4057a52a5ba90f316650"
format_stamp: "Formatted at 2026-06-19 08:54:54 on dist-test-slave-k50r"
I20260619 08:54:54.946686 404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 08:54:54.947620 418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:54.947777 404 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:54:54.947872 404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/wal
uuid: "571a7f3960cd4057a52a5ba90f316650"
format_stamp: "Formatted at 2026-06-19 08:54:54 on dist-test-slave-k50r"
I20260619 08:54:54.947962 404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:54:54.965626 404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:54:54.965905 404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:54:54.966064 404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:54:54.969898 404 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38311
I20260619 08:54:54.969928 470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38311 every 8 connection(s)
I20260619 08:54:54.970239 404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260619 08:54:54.970661 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXnrr3A/build/release/bin/kudu as pid 404
I20260619 08:54:54.970745 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260619 08:54:54.970991 471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:54:54.973640 471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 571a7f3960cd4057a52a5ba90f316650: Bootstrap starting.
I20260619 08:54:54.974231 471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 571a7f3960cd4057a52a5ba90f316650: Neither blocks nor log segments found. Creating new log.
I20260619 08:54:54.974565 471 log.cc:824] T 00000000000000000000000000000000 P 571a7f3960cd4057a52a5ba90f316650: Log is configured to *not* fsync() on all Append() calls
I20260619 08:54:54.975160 471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 571a7f3960cd4057a52a5ba90f316650: No bootstrap required, opened a new log
I20260619 08:54:54.976833 471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 571a7f3960cd4057a52a5ba90f316650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571a7f3960cd4057a52a5ba90f316650" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38311 } }
I20260619 08:54:54.976984 471 raft_consensus.cc:374] T 00000000000000000000000000000000 P 571a7f3960cd4057a52a5ba90f316650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:54:54.977049 471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 571a7f3960cd4057a52a5ba90f316650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 571a7f3960cd4057a52a5ba90f316650, State: Initialized, Role: FOLLOWER
I20260619 08:54:54.977195 471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 571a7f3960cd4057a52a5ba90f316650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571a7f3960cd4057a52a5ba90f316650" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38311 } }
I20260619 08:54:54.977301 471 raft_consensus.cc:388] T 00000000000000000000000000000000 P 571a7f3960cd4057a52a5ba90f316650 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:54:54.977342 471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 571a7f3960cd4057a52a5ba90f316650 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:54:54.977386 471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 571a7f3960cd4057a52a5ba90f316650 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:54:54.977847 471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 571a7f3960cd4057a52a5ba90f316650 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571a7f3960cd4057a52a5ba90f316650" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38311 } }
I20260619 08:54:54.977957 471 leader_election.cc:302] T 00000000000000000000000000000000 P 571a7f3960cd4057a52a5ba90f316650 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 571a7f3960cd4057a52a5ba90f316650; no voters:
I20260619 08:54:54.978132 471 leader_election.cc:288] T 00000000000000000000000000000000 P 571a7f3960cd4057a52a5ba90f316650 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:54:54.978161 476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 571a7f3960cd4057a52a5ba90f316650 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:54:54.978255 476 raft_consensus.cc:686] T 00000000000000000000000000000000 P 571a7f3960cd4057a52a5ba90f316650 [term 1 LEADER]: Becoming Leader. State: Replica: 571a7f3960cd4057a52a5ba90f316650, State: Running, Role: LEADER
I20260619 08:54:54.978344 476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 571a7f3960cd4057a52a5ba90f316650 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571a7f3960cd4057a52a5ba90f316650" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38311 } }
I20260619 08:54:54.978384 471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 571a7f3960cd4057a52a5ba90f316650 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:54:54.978688 476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 571a7f3960cd4057a52a5ba90f316650 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "571a7f3960cd4057a52a5ba90f316650" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571a7f3960cd4057a52a5ba90f316650" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38311 } } }
I20260619 08:54:54.978821 476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 571a7f3960cd4057a52a5ba90f316650 [sys.catalog]: This master's current role is: LEADER
I20260619 08:54:54.979027 476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 571a7f3960cd4057a52a5ba90f316650 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 571a7f3960cd4057a52a5ba90f316650. Latest consensus state: current_term: 1 leader_uuid: "571a7f3960cd4057a52a5ba90f316650" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571a7f3960cd4057a52a5ba90f316650" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38311 } } }
I20260619 08:54:54.979104 476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 571a7f3960cd4057a52a5ba90f316650 [sys.catalog]: This master's current role is: LEADER
I20260619 08:54:54.979413 483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:54:54.979918 483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:54:54.981482 483 catalog_manager.cc:1347] Generated new cluster ID: a603ee066bd54c21be404c31de5c1a6a
I20260619 08:54:54.981562 483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:54:54.996809 483 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:54:54.997356 483 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:54:55.006568 483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 571a7f3960cd4057a52a5ba90f316650: Generated new TSK 0
I20260619 08:54:55.006719 483 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:54:55.012359 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXnrr3A/build/release/bin/kudu
/tmp/dist-test-taskXnrr3A/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38311
--builtin_ntp_servers=127.0.98.20:42509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20260619 08:54:55.125986 495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:54:55.126264 495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:54:55.126295 495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:54:55.127894 495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:54:55.127969 495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:54:55.129832 495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38311
--never_fsync=true
--heap_profile_path=/tmp/kudu.495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:54:55.130069 495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:54:55.130363 495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:54:55.135241 500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:54:55.135248 501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:54:55.135248 503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:54:55.135774 495 server_base.cc:1034] running on GCE node
I20260619 08:54:55.135982 495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:54:55.136211 495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:54:55.137372 495 hybrid_clock.cc:648] HybridClock initialized: now 1781859295137351 us; error 40 us; skew 500 ppm
I20260619 08:54:55.138604 495 webserver.cc:460] Webserver started at http://127.0.98.1:45239/ using document root <none> and password file <none>
I20260619 08:54:55.138833 495 fs_manager.cc:359] Metadata directory not provided
I20260619 08:54:55.138921 495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:54:55.139062 495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:54:55.139976 495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "81a2f51aa77c419288af6ef289f812e1"
format_stamp: "Formatted at 2026-06-19 08:54:55 on dist-test-slave-k50r"
I20260619 08:54:55.140350 495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "81a2f51aa77c419288af6ef289f812e1"
format_stamp: "Formatted at 2026-06-19 08:54:55 on dist-test-slave-k50r"
I20260619 08:54:55.141686 495 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:54:55.142387 509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:55.142555 495 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:54:55.142650 495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "81a2f51aa77c419288af6ef289f812e1"
format_stamp: "Formatted at 2026-06-19 08:54:55 on dist-test-slave-k50r"
I20260619 08:54:55.142728 495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:54:55.161366 495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:54:55.161675 495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:54:55.161835 495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:54:55.162077 495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:54:55.162398 495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:54:55.162508 495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:55.162547 495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:54:55.162576 495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:55.168717 495 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46201
I20260619 08:54:55.168792 622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46201 every 8 connection(s)
I20260619 08:54:55.169126 495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260619 08:54:55.174381 623 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38311
I20260619 08:54:55.174494 623 heartbeater.cc:464] Registering TS with master...
I20260619 08:54:55.174654 623 heartbeater.cc:511] Master 127.0.98.62:38311 requested a full tablet report, sending...
I20260619 08:54:55.175071 435 ts_manager.cc:194] Registered new tserver with Master: 81a2f51aa77c419288af6ef289f812e1 (127.0.98.1:46201)
I20260619 08:54:55.175740 435 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:34479
I20260619 08:54:55.177994 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXnrr3A/build/release/bin/kudu as pid 495
I20260619 08:54:55.178077 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260619 08:54:55.179253 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXnrr3A/build/release/bin/kudu
/tmp/dist-test-taskXnrr3A/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38311
--builtin_ntp_servers=127.0.98.20:42509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20260619 08:54:55.261816 626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:54:55.261997 626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:54:55.262046 626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:54:55.263832 626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:54:55.263926 626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:54:55.265206 626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38311
--never_fsync=true
--heap_profile_path=/tmp/kudu.626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:54:55.265445 626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:54:55.265753 626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:54:55.270604 631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:54:55.270677 632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:54:55.270629 634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:54:55.270781 626 server_base.cc:1034] running on GCE node
I20260619 08:54:55.271034 626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:54:55.271313 626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:54:55.272471 626 hybrid_clock.cc:648] HybridClock initialized: now 1781859295272451 us; error 33 us; skew 500 ppm
I20260619 08:54:55.273669 626 webserver.cc:460] Webserver started at http://127.0.98.2:32777/ using document root <none> and password file <none>
I20260619 08:54:55.273857 626 fs_manager.cc:359] Metadata directory not provided
I20260619 08:54:55.273897 626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:54:55.273980 626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:54:55.275038 626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "566e3f52254b461382fecf40daa72ccf"
format_stamp: "Formatted at 2026-06-19 08:54:55 on dist-test-slave-k50r"
I20260619 08:54:55.275331 626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "566e3f52254b461382fecf40daa72ccf"
format_stamp: "Formatted at 2026-06-19 08:54:55 on dist-test-slave-k50r"
I20260619 08:54:55.276630 626 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:54:55.277256 640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:55.277437 626 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:54:55.277504 626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "566e3f52254b461382fecf40daa72ccf"
format_stamp: "Formatted at 2026-06-19 08:54:55 on dist-test-slave-k50r"
I20260619 08:54:55.277585 626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:54:55.287043 626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:54:55.287295 626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:54:55.287434 626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:54:55.287642 626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:54:55.287919 626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:54:55.287971 626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:55.288008 626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:54:55.288036 626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:55.293838 626 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38661
I20260619 08:54:55.293921 753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38661 every 8 connection(s)
I20260619 08:54:55.294224 626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260619 08:54:55.299250 754 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38311
I20260619 08:54:55.299326 754 heartbeater.cc:464] Registering TS with master...
I20260619 08:54:55.299508 754 heartbeater.cc:511] Master 127.0.98.62:38311 requested a full tablet report, sending...
I20260619 08:54:55.299829 435 ts_manager.cc:194] Registered new tserver with Master: 566e3f52254b461382fecf40daa72ccf (127.0.98.2:38661)
I20260619 08:54:55.300210 435 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:46287
I20260619 08:54:55.303682 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXnrr3A/build/release/bin/kudu as pid 626
I20260619 08:54:55.303755 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260619 08:54:55.304733 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXnrr3A/build/release/bin/kudu
/tmp/dist-test-taskXnrr3A/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38311
--builtin_ntp_servers=127.0.98.20:42509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20260619 08:54:55.383653 757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:54:55.383900 757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:54:55.383936 757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:54:55.385064 757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:54:55.385149 757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:54:55.386456 757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38311
--never_fsync=true
--heap_profile_path=/tmp/kudu.757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:54:55.386696 757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:54:55.386969 757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:54:55.393182 762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:54:55.393172 763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:54:55.393802 757 server_base.cc:1034] running on GCE node
W20260619 08:54:55.394626 765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:54:55.394884 757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:54:55.395186 757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:54:55.396334 757 hybrid_clock.cc:648] HybridClock initialized: now 1781859295396317 us; error 32 us; skew 500 ppm
I20260619 08:54:55.397789 757 webserver.cc:460] Webserver started at http://127.0.98.3:35885/ using document root <none> and password file <none>
I20260619 08:54:55.397972 757 fs_manager.cc:359] Metadata directory not provided
I20260619 08:54:55.398012 757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:54:55.398098 757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:54:55.399071 757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "243f8313fe1f4d6f805f0acb4139c7f9"
format_stamp: "Formatted at 2026-06-19 08:54:55 on dist-test-slave-k50r"
I20260619 08:54:55.399380 757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "243f8313fe1f4d6f805f0acb4139c7f9"
format_stamp: "Formatted at 2026-06-19 08:54:55 on dist-test-slave-k50r"
I20260619 08:54:55.400578 757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 08:54:55.401186 771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:55.401347 757 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:54:55.401449 757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "243f8313fe1f4d6f805f0acb4139c7f9"
format_stamp: "Formatted at 2026-06-19 08:54:55 on dist-test-slave-k50r"
I20260619 08:54:55.401532 757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:54:55.424355 757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:54:55.424623 757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:54:55.424772 757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:54:55.424999 757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:54:55.425295 757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:54:55.425346 757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:55.425465 757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:54:55.425495 757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:55.431737 757 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39859
I20260619 08:54:55.431780 884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39859 every 8 connection(s)
I20260619 08:54:55.432070 757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260619 08:54:55.437520 885 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38311
I20260619 08:54:55.437604 885 heartbeater.cc:464] Registering TS with master...
I20260619 08:54:55.437736 885 heartbeater.cc:511] Master 127.0.98.62:38311 requested a full tablet report, sending...
I20260619 08:54:55.438072 435 ts_manager.cc:194] Registered new tserver with Master: 243f8313fe1f4d6f805f0acb4139c7f9 (127.0.98.3:39859)
I20260619 08:54:55.438397 435 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:33177
I20260619 08:54:55.439226 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXnrr3A/build/release/bin/kudu as pid 757
I20260619 08:54:55.439304 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260619 08:54:55.440426 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:54:55.446900 435 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40688:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260619 08:54:55.448598 435 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 08:54:55.455430 557 tablet_service.cc:1467] Processing CreateTablet for tablet 048527e37f4b48829a84024954cc11f4 (DEFAULT_TABLE table=TestTable [id=6444dd73a23f450dad6460acf2cc610b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:54:55.455792 557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 048527e37f4b48829a84024954cc11f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:54:55.455685 688 tablet_service.cc:1467] Processing CreateTablet for tablet 048527e37f4b48829a84024954cc11f4 (DEFAULT_TABLE table=TestTable [id=6444dd73a23f450dad6460acf2cc610b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:54:55.456036 688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 048527e37f4b48829a84024954cc11f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:54:55.458050 899 tablet_bootstrap.cc:492] T 048527e37f4b48829a84024954cc11f4 P 81a2f51aa77c419288af6ef289f812e1: Bootstrap starting.
I20260619 08:54:55.458098 900 tablet_bootstrap.cc:492] T 048527e37f4b48829a84024954cc11f4 P 566e3f52254b461382fecf40daa72ccf: Bootstrap starting.
I20260619 08:54:55.458663 819 tablet_service.cc:1467] Processing CreateTablet for tablet 048527e37f4b48829a84024954cc11f4 (DEFAULT_TABLE table=TestTable [id=6444dd73a23f450dad6460acf2cc610b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:54:55.458782 899 tablet_bootstrap.cc:654] T 048527e37f4b48829a84024954cc11f4 P 81a2f51aa77c419288af6ef289f812e1: Neither blocks nor log segments found. Creating new log.
I20260619 08:54:55.458797 900 tablet_bootstrap.cc:654] T 048527e37f4b48829a84024954cc11f4 P 566e3f52254b461382fecf40daa72ccf: Neither blocks nor log segments found. Creating new log.
I20260619 08:54:55.458983 819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 048527e37f4b48829a84024954cc11f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:54:55.459103 899 log.cc:824] T 048527e37f4b48829a84024954cc11f4 P 81a2f51aa77c419288af6ef289f812e1: Log is configured to *not* fsync() on all Append() calls
I20260619 08:54:55.459126 900 log.cc:824] T 048527e37f4b48829a84024954cc11f4 P 566e3f52254b461382fecf40daa72ccf: Log is configured to *not* fsync() on all Append() calls
I20260619 08:54:55.459805 899 tablet_bootstrap.cc:492] T 048527e37f4b48829a84024954cc11f4 P 81a2f51aa77c419288af6ef289f812e1: No bootstrap required, opened a new log
I20260619 08:54:55.459832 900 tablet_bootstrap.cc:492] T 048527e37f4b48829a84024954cc11f4 P 566e3f52254b461382fecf40daa72ccf: No bootstrap required, opened a new log
I20260619 08:54:55.459873 899 ts_tablet_manager.cc:1397] T 048527e37f4b48829a84024954cc11f4 P 81a2f51aa77c419288af6ef289f812e1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 08:54:55.459910 900 ts_tablet_manager.cc:1397] T 048527e37f4b48829a84024954cc11f4 P 566e3f52254b461382fecf40daa72ccf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 08:54:55.462078 900 raft_consensus.cc:348] T 048527e37f4b48829a84024954cc11f4 P 566e3f52254b461382fecf40daa72ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "243f8313fe1f4d6f805f0acb4139c7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39859 } } peers { permanent_uuid: "566e3f52254b461382fecf40daa72ccf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38661 } } peers { permanent_uuid: "81a2f51aa77c419288af6ef289f812e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46201 } }
I20260619 08:54:55.462244 900 raft_consensus.cc:374] T 048527e37f4b48829a84024954cc11f4 P 566e3f52254b461382fecf40daa72ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:54:55.462253 903 tablet_bootstrap.cc:492] T 048527e37f4b48829a84024954cc11f4 P 243f8313fe1f4d6f805f0acb4139c7f9: Bootstrap starting.
I20260619 08:54:55.462285 900 raft_consensus.cc:729] T 048527e37f4b48829a84024954cc11f4 P 566e3f52254b461382fecf40daa72ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 566e3f52254b461382fecf40daa72ccf, State: Initialized, Role: FOLLOWER
I20260619 08:54:55.462440 900 consensus_queue.cc:260] T 048527e37f4b48829a84024954cc11f4 P 566e3f52254b461382fecf40daa72ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "243f8313fe1f4d6f805f0acb4139c7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39859 } } peers { permanent_uuid: "566e3f52254b461382fecf40daa72ccf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38661 } } peers { permanent_uuid: "81a2f51aa77c419288af6ef289f812e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46201 } }
I20260619 08:54:55.462402 899 raft_consensus.cc:348] T 048527e37f4b48829a84024954cc11f4 P 81a2f51aa77c419288af6ef289f812e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "243f8313fe1f4d6f805f0acb4139c7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39859 } } peers { permanent_uuid: "566e3f52254b461382fecf40daa72ccf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38661 } } peers { permanent_uuid: "81a2f51aa77c419288af6ef289f812e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46201 } }
I20260619 08:54:55.462608 899 raft_consensus.cc:374] T 048527e37f4b48829a84024954cc11f4 P 81a2f51aa77c419288af6ef289f812e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:54:55.462656 899 raft_consensus.cc:729] T 048527e37f4b48829a84024954cc11f4 P 81a2f51aa77c419288af6ef289f812e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81a2f51aa77c419288af6ef289f812e1, State: Initialized, Role: FOLLOWER
I20260619 08:54:55.462700 900 ts_tablet_manager.cc:1428] T 048527e37f4b48829a84024954cc11f4 P 566e3f52254b461382fecf40daa72ccf: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260619 08:54:55.462814 754 heartbeater.cc:503] Master 127.0.98.62:38311 was elected leader, sending a full tablet report...
I20260619 08:54:55.462801 899 consensus_queue.cc:260] T 048527e37f4b48829a84024954cc11f4 P 81a2f51aa77c419288af6ef289f812e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "243f8313fe1f4d6f805f0acb4139c7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39859 } } peers { permanent_uuid: "566e3f52254b461382fecf40daa72ccf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38661 } } peers { permanent_uuid: "81a2f51aa77c419288af6ef289f812e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46201 } }
I20260619 08:54:55.462919 903 tablet_bootstrap.cc:654] T 048527e37f4b48829a84024954cc11f4 P 243f8313fe1f4d6f805f0acb4139c7f9: Neither blocks nor log segments found. Creating new log.
I20260619 08:54:55.463078 899 ts_tablet_manager.cc:1428] T 048527e37f4b48829a84024954cc11f4 P 81a2f51aa77c419288af6ef289f812e1: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 08:54:55.463145 623 heartbeater.cc:503] Master 127.0.98.62:38311 was elected leader, sending a full tablet report...
I20260619 08:54:55.463208 903 log.cc:824] T 048527e37f4b48829a84024954cc11f4 P 243f8313fe1f4d6f805f0acb4139c7f9: Log is configured to *not* fsync() on all Append() calls
I20260619 08:54:55.463847 903 tablet_bootstrap.cc:492] T 048527e37f4b48829a84024954cc11f4 P 243f8313fe1f4d6f805f0acb4139c7f9: No bootstrap required, opened a new log
I20260619 08:54:55.463948 903 ts_tablet_manager.cc:1397] T 048527e37f4b48829a84024954cc11f4 P 243f8313fe1f4d6f805f0acb4139c7f9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 08:54:55.465528 903 raft_consensus.cc:348] T 048527e37f4b48829a84024954cc11f4 P 243f8313fe1f4d6f805f0acb4139c7f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "243f8313fe1f4d6f805f0acb4139c7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39859 } } peers { permanent_uuid: "566e3f52254b461382fecf40daa72ccf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38661 } } peers { permanent_uuid: "81a2f51aa77c419288af6ef289f812e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46201 } }
I20260619 08:54:55.465636 903 raft_consensus.cc:374] T 048527e37f4b48829a84024954cc11f4 P 243f8313fe1f4d6f805f0acb4139c7f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:54:55.465656 903 raft_consensus.cc:729] T 048527e37f4b48829a84024954cc11f4 P 243f8313fe1f4d6f805f0acb4139c7f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 243f8313fe1f4d6f805f0acb4139c7f9, State: Initialized, Role: FOLLOWER
I20260619 08:54:55.465785 903 consensus_queue.cc:260] T 048527e37f4b48829a84024954cc11f4 P 243f8313fe1f4d6f805f0acb4139c7f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "243f8313fe1f4d6f805f0acb4139c7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39859 } } peers { permanent_uuid: "566e3f52254b461382fecf40daa72ccf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38661 } } peers { permanent_uuid: "81a2f51aa77c419288af6ef289f812e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46201 } }
I20260619 08:54:55.466033 903 ts_tablet_manager.cc:1428] T 048527e37f4b48829a84024954cc11f4 P 243f8313fe1f4d6f805f0acb4139c7f9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 08:54:55.466055 885 heartbeater.cc:503] Master 127.0.98.62:38311 was elected leader, sending a full tablet report...
I20260619 08:54:55.544128 905 raft_consensus.cc:482] T 048527e37f4b48829a84024954cc11f4 P 81a2f51aa77c419288af6ef289f812e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:54:55.544234 905 raft_consensus.cc:504] T 048527e37f4b48829a84024954cc11f4 P 81a2f51aa77c419288af6ef289f812e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "243f8313fe1f4d6f805f0acb4139c7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39859 } } peers { permanent_uuid: "566e3f52254b461382fecf40daa72ccf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38661 } } peers { permanent_uuid: "81a2f51aa77c419288af6ef289f812e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46201 } }
W20260619 08:54:55.544816 755 tablet.cc:2369] T 048527e37f4b48829a84024954cc11f4 P 566e3f52254b461382fecf40daa72ccf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:54:55.544826 905 leader_election.cc:288] T 048527e37f4b48829a84024954cc11f4 P 81a2f51aa77c419288af6ef289f812e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 243f8313fe1f4d6f805f0acb4139c7f9 (127.0.98.3:39859), 566e3f52254b461382fecf40daa72ccf (127.0.98.2:38661)
I20260619 08:54:55.548072 839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "048527e37f4b48829a84024954cc11f4" candidate_uuid: "81a2f51aa77c419288af6ef289f812e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "243f8313fe1f4d6f805f0acb4139c7f9" is_pre_election: true
I20260619 08:54:55.548254 839 raft_consensus.cc:2454] T 048527e37f4b48829a84024954cc11f4 P 243f8313fe1f4d6f805f0acb4139c7f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a2f51aa77c419288af6ef289f812e1 in term 0.
I20260619 08:54:55.548220 708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "048527e37f4b48829a84024954cc11f4" candidate_uuid: "81a2f51aa77c419288af6ef289f812e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "566e3f52254b461382fecf40daa72ccf" is_pre_election: true
I20260619 08:54:55.548333 708 raft_consensus.cc:2454] T 048527e37f4b48829a84024954cc11f4 P 566e3f52254b461382fecf40daa72ccf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a2f51aa77c419288af6ef289f812e1 in term 0.
I20260619 08:54:55.548497 511 leader_election.cc:302] T 048527e37f4b48829a84024954cc11f4 P 81a2f51aa77c419288af6ef289f812e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 243f8313fe1f4d6f805f0acb4139c7f9, 81a2f51aa77c419288af6ef289f812e1; no voters:
I20260619 08:54:55.548640 905 raft_consensus.cc:2781] T 048527e37f4b48829a84024954cc11f4 P 81a2f51aa77c419288af6ef289f812e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:54:55.548712 905 raft_consensus.cc:482] T 048527e37f4b48829a84024954cc11f4 P 81a2f51aa77c419288af6ef289f812e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:54:55.548730 905 raft_consensus.cc:3037] T 048527e37f4b48829a84024954cc11f4 P 81a2f51aa77c419288af6ef289f812e1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:54:55.549460 905 raft_consensus.cc:504] T 048527e37f4b48829a84024954cc11f4 P 81a2f51aa77c419288af6ef289f812e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "243f8313fe1f4d6f805f0acb4139c7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39859 } } peers { permanent_uuid: "566e3f52254b461382fecf40daa72ccf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38661 } } peers { permanent_uuid: "81a2f51aa77c419288af6ef289f812e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46201 } }
I20260619 08:54:55.549592 905 leader_election.cc:288] T 048527e37f4b48829a84024954cc11f4 P 81a2f51aa77c419288af6ef289f812e1 [CANDIDATE]: Term 1 election: Requested vote from peers 243f8313fe1f4d6f805f0acb4139c7f9 (127.0.98.3:39859), 566e3f52254b461382fecf40daa72ccf (127.0.98.2:38661)
I20260619 08:54:55.549733 708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "048527e37f4b48829a84024954cc11f4" candidate_uuid: "81a2f51aa77c419288af6ef289f812e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "566e3f52254b461382fecf40daa72ccf"
I20260619 08:54:55.549851 708 raft_consensus.cc:3037] T 048527e37f4b48829a84024954cc11f4 P 566e3f52254b461382fecf40daa72ccf [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:54:55.550594 708 raft_consensus.cc:2454] T 048527e37f4b48829a84024954cc11f4 P 566e3f52254b461382fecf40daa72ccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a2f51aa77c419288af6ef289f812e1 in term 1.
I20260619 08:54:55.550714 839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "048527e37f4b48829a84024954cc11f4" candidate_uuid: "81a2f51aa77c419288af6ef289f812e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "243f8313fe1f4d6f805f0acb4139c7f9"
I20260619 08:54:55.550767 510 leader_election.cc:302] T 048527e37f4b48829a84024954cc11f4 P 81a2f51aa77c419288af6ef289f812e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 566e3f52254b461382fecf40daa72ccf, 81a2f51aa77c419288af6ef289f812e1; no voters:
I20260619 08:54:55.550792 839 raft_consensus.cc:3037] T 048527e37f4b48829a84024954cc11f4 P 243f8313fe1f4d6f805f0acb4139c7f9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:54:55.550872 905 raft_consensus.cc:2781] T 048527e37f4b48829a84024954cc11f4 P 81a2f51aa77c419288af6ef289f812e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:54:55.551039 905 raft_consensus.cc:686] T 048527e37f4b48829a84024954cc11f4 P 81a2f51aa77c419288af6ef289f812e1 [term 1 LEADER]: Becoming Leader. State: Replica: 81a2f51aa77c419288af6ef289f812e1, State: Running, Role: LEADER
I20260619 08:54:55.551126 905 consensus_queue.cc:237] T 048527e37f4b48829a84024954cc11f4 P 81a2f51aa77c419288af6ef289f812e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "243f8313fe1f4d6f805f0acb4139c7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39859 } } peers { permanent_uuid: "566e3f52254b461382fecf40daa72ccf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38661 } } peers { permanent_uuid: "81a2f51aa77c419288af6ef289f812e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46201 } }
I20260619 08:54:55.551630 839 raft_consensus.cc:2454] T 048527e37f4b48829a84024954cc11f4 P 243f8313fe1f4d6f805f0acb4139c7f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a2f51aa77c419288af6ef289f812e1 in term 1.
I20260619 08:54:55.552151 435 catalog_manager.cc:5495] T 048527e37f4b48829a84024954cc11f4 P 81a2f51aa77c419288af6ef289f812e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81a2f51aa77c419288af6ef289f812e1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "81a2f51aa77c419288af6ef289f812e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "243f8313fe1f4d6f805f0acb4139c7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "566e3f52254b461382fecf40daa72ccf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81a2f51aa77c419288af6ef289f812e1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46201 } health_report { overall_health: HEALTHY } } }
I20260619 08:54:55.566875 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 08:54:55.567548 392 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 81a2f51aa77c419288af6ef289f812e1 to finish bootstrapping
I20260619 08:54:55.571566 392 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 566e3f52254b461382fecf40daa72ccf to finish bootstrapping
I20260619 08:54:55.574721 392 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 243f8313fe1f4d6f805f0acb4139c7f9 to finish bootstrapping
I20260619 08:54:55.587920 392 cluster_itest_util.cc:256] Not converged past 1 yet: 0.0 0.0 1.1
W20260619 08:54:55.669818 624 tablet.cc:2369] T 048527e37f4b48829a84024954cc11f4 P 81a2f51aa77c419288af6ef289f812e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:54:55.682623 886 tablet.cc:2369] T 048527e37f4b48829a84024954cc11f4 P 243f8313fe1f4d6f805f0acb4139c7f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:54:55.689414 392 cluster_itest_util.cc:256] Not converged past 1 yet: 0.0 0.0 1.1
I20260619 08:54:55.890848 392 cluster_itest_util.cc:256] Not converged past 1 yet: 0.0 0.0 1.1
I20260619 08:54:55.961586 905 consensus_queue.cc:1035] T 048527e37f4b48829a84024954cc11f4 P 81a2f51aa77c419288af6ef289f812e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "243f8313fe1f4d6f805f0acb4139c7f9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:54:55.963922 915 consensus_queue.cc:1035] T 048527e37f4b48829a84024954cc11f4 P 81a2f51aa77c419288af6ef289f812e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "566e3f52254b461382fecf40daa72ccf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 08:54:56.194514 914 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43628: Not implemented: Unknown operation type: 115
I20260619 08:54:56.225852 688 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260619 08:54:56.225924 819 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20260619 08:54:56.225920 557 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
571a7f3960cd4057a52a5ba90f316650 | 127.0.98.62:38311 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.98.20:42509 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
243f8313fe1f4d6f805f0acb4139c7f9 | 127.0.98.3:39859 | HEALTHY | <none> | 0 | 0
566e3f52254b461382fecf40daa72ccf | 127.0.98.2:38661 | HEALTHY | <none> | 0 | 0
81a2f51aa77c419288af6ef289f812e1 | 127.0.98.1:46201 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.0.98.1 | experimental | 127.0.98.1:46201
local_ip_for_outbound_sockets | 127.0.98.2 | experimental | 127.0.98.2:38661
local_ip_for_outbound_sockets | 127.0.98.3 | experimental | 127.0.98.3:39859
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.98.1:46201
server_dump_info_path | /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.98.2:38661
server_dump_info_path | /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.TestFailedOp.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.98.3:39859
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.0.98.20:42509 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260619 08:54:56.236426 392 log_verifier.cc:126] Checking tablet 048527e37f4b48829a84024954cc11f4
I20260619 08:54:56.254702 392 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 048527e37f4b48829a84024954cc11f4
I20260619 08:54:56.266462 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXnrr3A/build/release/bin/kudu with pid 495
I20260619 08:54:56.272382 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXnrr3A/build/release/bin/kudu with pid 626
I20260619 08:54:56.278791 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXnrr3A/build/release/bin/kudu with pid 757
I20260619 08:54:56.283380 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskXnrr3A/build/release/bin/kudu with pid 404
2026-06-19T08:54:56Z chronyd exiting
[ OK ] RaftConsensusITest.TestFailedOp (1462 ms)
[ RUN ] RaftConsensusITest.MultiThreadedInsertWithFailovers
I20260619 08:54:56.304591 392 test_util.cc:268] Using random seed: 1919245772
I20260619 08:54:56.305135 392 ts_itest-base.cc:112] Starting cluster with:
I20260619 08:54:56.305184 392 ts_itest-base.cc:113] --------------
I20260619 08:54:56.305204 392 ts_itest-base.cc:114] 7 tablet servers
I20260619 08:54:56.305243 392 ts_itest-base.cc:115] 7 replicas per TS
I20260619 08:54:56.305263 392 ts_itest-base.cc:116] --------------
2026-06-19T08:54:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T08:54:56Z Disabled control of system clock
I20260619 08:54:56.312357 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXnrr3A/build/release/bin/kudu
/tmp/dist-test-taskXnrr3A/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43691
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43691 with env {}
W20260619 08:54:56.389160 965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:54:56.389394 965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:54:56.389427 965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:54:56.390558 965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 08:54:56.390623 965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:54:56.390643 965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 08:54:56.390663 965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 08:54:56.391971 965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43691
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:54:56.392220 965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:54:56.392490 965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:54:56.397192 973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:54:56.397202 971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:54:56.397449 965 server_base.cc:1034] running on GCE node
W20260619 08:54:56.397236 970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:54:56.397794 965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:54:56.398135 965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:54:56.399318 965 hybrid_clock.cc:648] HybridClock initialized: now 1781859296399294 us; error 38 us; skew 500 ppm
I20260619 08:54:56.400626 965 webserver.cc:460] Webserver started at http://127.0.98.62:36599/ using document root <none> and password file <none>
I20260619 08:54:56.400880 965 fs_manager.cc:359] Metadata directory not provided
I20260619 08:54:56.400954 965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:54:56.401096 965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:54:56.402067 965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3dc87e65ba584f27bba202235b1600da"
format_stamp: "Formatted at 2026-06-19 08:54:56 on dist-test-slave-k50r"
I20260619 08:54:56.402495 965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3dc87e65ba584f27bba202235b1600da"
format_stamp: "Formatted at 2026-06-19 08:54:56 on dist-test-slave-k50r"
I20260619 08:54:56.403801 965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 08:54:56.404623 979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:56.404795 965 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:54:56.404894 965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3dc87e65ba584f27bba202235b1600da"
format_stamp: "Formatted at 2026-06-19 08:54:56 on dist-test-slave-k50r"
I20260619 08:54:56.405006 965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:54:56.433442 965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:54:56.433746 965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:54:56.433899 965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:54:56.437675 965 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43691
I20260619 08:54:56.437727 1031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43691 every 8 connection(s)
I20260619 08:54:56.438094 965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20260619 08:54:56.438747 1032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:54:56.441002 1032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dc87e65ba584f27bba202235b1600da: Bootstrap starting.
I20260619 08:54:56.441488 1032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3dc87e65ba584f27bba202235b1600da: Neither blocks nor log segments found. Creating new log.
I20260619 08:54:56.441699 1032 log.cc:824] T 00000000000000000000000000000000 P 3dc87e65ba584f27bba202235b1600da: Log is configured to *not* fsync() on all Append() calls
I20260619 08:54:56.442288 1032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dc87e65ba584f27bba202235b1600da: No bootstrap required, opened a new log
I20260619 08:54:56.443856 1032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dc87e65ba584f27bba202235b1600da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc87e65ba584f27bba202235b1600da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43691 } }
I20260619 08:54:56.443981 1032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3dc87e65ba584f27bba202235b1600da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:54:56.444005 1032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dc87e65ba584f27bba202235b1600da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc87e65ba584f27bba202235b1600da, State: Initialized, Role: FOLLOWER
I20260619 08:54:56.444131 1032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dc87e65ba584f27bba202235b1600da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc87e65ba584f27bba202235b1600da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43691 } }
I20260619 08:54:56.444207 1032 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dc87e65ba584f27bba202235b1600da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 08:54:56.444229 1032 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dc87e65ba584f27bba202235b1600da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 08:54:56.444255 1032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dc87e65ba584f27bba202235b1600da [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:54:56.444662 1032 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dc87e65ba584f27bba202235b1600da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc87e65ba584f27bba202235b1600da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43691 } }
I20260619 08:54:56.444737 1032 leader_election.cc:302] T 00000000000000000000000000000000 P 3dc87e65ba584f27bba202235b1600da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dc87e65ba584f27bba202235b1600da; no voters:
I20260619 08:54:56.444855 1032 leader_election.cc:288] T 00000000000000000000000000000000 P 3dc87e65ba584f27bba202235b1600da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 08:54:56.444918 1035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dc87e65ba584f27bba202235b1600da [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:54:56.445066 1035 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dc87e65ba584f27bba202235b1600da [term 1 LEADER]: Becoming Leader. State: Replica: 3dc87e65ba584f27bba202235b1600da, State: Running, Role: LEADER
I20260619 08:54:56.445091 1032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dc87e65ba584f27bba202235b1600da [sys.catalog]: configured and running, proceeding with master startup.
I20260619 08:54:56.445158 1035 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dc87e65ba584f27bba202235b1600da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc87e65ba584f27bba202235b1600da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43691 } }
I20260619 08:54:56.445511 1036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dc87e65ba584f27bba202235b1600da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3dc87e65ba584f27bba202235b1600da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc87e65ba584f27bba202235b1600da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43691 } } }
I20260619 08:54:56.445519 1037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dc87e65ba584f27bba202235b1600da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dc87e65ba584f27bba202235b1600da. Latest consensus state: current_term: 1 leader_uuid: "3dc87e65ba584f27bba202235b1600da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc87e65ba584f27bba202235b1600da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43691 } } }
I20260619 08:54:56.445605 1036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dc87e65ba584f27bba202235b1600da [sys.catalog]: This master's current role is: LEADER
I20260619 08:54:56.445611 1037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dc87e65ba584f27bba202235b1600da [sys.catalog]: This master's current role is: LEADER
I20260619 08:54:56.445809 1044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 08:54:56.446266 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXnrr3A/build/release/bin/kudu as pid 965
I20260619 08:54:56.446353 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/master-0/wal/instance
I20260619 08:54:56.446847 1044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 08:54:56.448418 1044 catalog_manager.cc:1347] Generated new cluster ID: 9582cbe727ac4efe97c30586122aa247
I20260619 08:54:56.448468 1044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 08:54:56.454506 1044 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 08:54:56.455185 1044 catalog_manager.cc:1504] Loading token signing keys...
I20260619 08:54:56.462055 1044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3dc87e65ba584f27bba202235b1600da: Generated new TSK 0
I20260619 08:54:56.462237 1044 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 08:54:56.465754 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXnrr3A/build/release/bin/kudu
/tmp/dist-test-taskXnrr3A/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43691
--builtin_ntp_servers=127.0.98.20:35761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
W20260619 08:54:56.544772 1056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:54:56.544960 1056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:54:56.544981 1056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:54:56.546252 1056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:54:56.546306 1056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 08:54:56.547719 1056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43691
--never_fsync=true
--heap_profile_path=/tmp/kudu.1056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:54:56.547955 1056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:54:56.548249 1056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:54:56.553057 1061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:54:56.553045 1064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:54:56.553045 1062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:54:56.553789 1056 server_base.cc:1034] running on GCE node
I20260619 08:54:56.554018 1056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:54:56.554263 1056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:54:56.555435 1056 hybrid_clock.cc:648] HybridClock initialized: now 1781859296555424 us; error 35 us; skew 500 ppm
I20260619 08:54:56.556645 1056 webserver.cc:460] Webserver started at http://127.0.98.1:46349/ using document root <none> and password file <none>
I20260619 08:54:56.556883 1056 fs_manager.cc:359] Metadata directory not provided
I20260619 08:54:56.556953 1056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:54:56.557085 1056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:54:56.558029 1056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "83f2ab318c954848b2be15d0926b94fb"
format_stamp: "Formatted at 2026-06-19 08:54:56 on dist-test-slave-k50r"
I20260619 08:54:56.558393 1056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "83f2ab318c954848b2be15d0926b94fb"
format_stamp: "Formatted at 2026-06-19 08:54:56 on dist-test-slave-k50r"
I20260619 08:54:56.559789 1056 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:54:56.560563 1070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:56.560806 1056 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 08:54:56.560909 1056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "83f2ab318c954848b2be15d0926b94fb"
format_stamp: "Formatted at 2026-06-19 08:54:56 on dist-test-slave-k50r"
I20260619 08:54:56.560979 1056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:54:56.583549 1056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:54:56.583784 1056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:54:56.583889 1056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:54:56.584060 1056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:54:56.584306 1056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:54:56.584332 1056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:56.584353 1056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:54:56.584365 1056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:56.589915 1056 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39977
I20260619 08:54:56.589979 1183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39977 every 8 connection(s)
I20260619 08:54:56.590322 1056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20260619 08:54:56.595981 1184 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43691
I20260619 08:54:56.596083 1184 heartbeater.cc:464] Registering TS with master...
I20260619 08:54:56.596242 1184 heartbeater.cc:511] Master 127.0.98.62:43691 requested a full tablet report, sending...
I20260619 08:54:56.596771 996 ts_manager.cc:194] Registered new tserver with Master: 83f2ab318c954848b2be15d0926b94fb (127.0.98.1:39977)
I20260619 08:54:56.597373 996 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:57697
I20260619 08:54:56.600361 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXnrr3A/build/release/bin/kudu as pid 1056
I20260619 08:54:56.600426 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20260619 08:54:56.601455 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXnrr3A/build/release/bin/kudu
/tmp/dist-test-taskXnrr3A/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43691
--builtin_ntp_servers=127.0.98.20:35761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
W20260619 08:54:56.678781 1187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:54:56.678977 1187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:54:56.678995 1187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:54:56.680141 1187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:54:56.680189 1187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 08:54:56.681429 1187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43691
--never_fsync=true
--heap_profile_path=/tmp/kudu.1187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:54:56.681691 1187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:54:56.681964 1187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:54:56.686816 1195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:54:56.686857 1192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:54:56.687049 1187 server_base.cc:1034] running on GCE node
W20260619 08:54:56.686887 1193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:54:56.687330 1187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:54:56.687525 1187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:54:56.688666 1187 hybrid_clock.cc:648] HybridClock initialized: now 1781859296688648 us; error 42 us; skew 500 ppm
I20260619 08:54:56.689859 1187 webserver.cc:460] Webserver started at http://127.0.98.2:44975/ using document root <none> and password file <none>
I20260619 08:54:56.690081 1187 fs_manager.cc:359] Metadata directory not provided
I20260619 08:54:56.690160 1187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:54:56.690318 1187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:54:56.691326 1187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "408d1f4ba9f245f1aa2608445e067a01"
format_stamp: "Formatted at 2026-06-19 08:54:56 on dist-test-slave-k50r"
I20260619 08:54:56.691658 1187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "408d1f4ba9f245f1aa2608445e067a01"
format_stamp: "Formatted at 2026-06-19 08:54:56 on dist-test-slave-k50r"
I20260619 08:54:56.692862 1187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 08:54:56.693542 1201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:56.693708 1187 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 08:54:56.693809 1187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "408d1f4ba9f245f1aa2608445e067a01"
format_stamp: "Formatted at 2026-06-19 08:54:56 on dist-test-slave-k50r"
I20260619 08:54:56.693887 1187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:54:56.712385 1187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:54:56.712654 1187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:54:56.712800 1187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:54:56.713008 1187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:54:56.713311 1187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:54:56.713367 1187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:56.713406 1187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:54:56.713433 1187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:56.719069 1187 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46549
I20260619 08:54:56.719126 1314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46549 every 8 connection(s)
I20260619 08:54:56.719449 1187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20260619 08:54:56.724977 1315 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43691
I20260619 08:54:56.725064 1315 heartbeater.cc:464] Registering TS with master...
I20260619 08:54:56.725257 1315 heartbeater.cc:511] Master 127.0.98.62:43691 requested a full tablet report, sending...
I20260619 08:54:56.725675 996 ts_manager.cc:194] Registered new tserver with Master: 408d1f4ba9f245f1aa2608445e067a01 (127.0.98.2:46549)
I20260619 08:54:56.725822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXnrr3A/build/release/bin/kudu as pid 1187
I20260619 08:54:56.725924 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20260619 08:54:56.726128 996 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:35053
I20260619 08:54:56.726925 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXnrr3A/build/release/bin/kudu
/tmp/dist-test-taskXnrr3A/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43691
--builtin_ntp_servers=127.0.98.20:35761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
W20260619 08:54:56.803857 1318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:54:56.804037 1318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:54:56.804056 1318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:54:56.805238 1318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:54:56.805307 1318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 08:54:56.806620 1318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43691
--never_fsync=true
--heap_profile_path=/tmp/kudu.1318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:54:56.806846 1318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:54:56.807133 1318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 08:54:56.811962 1318 server_base.cc:1034] running on GCE node
W20260619 08:54:56.812049 1324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:54:56.812016 1323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:54:56.812163 1326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:54:56.812359 1318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:54:56.812549 1318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:54:56.813695 1318 hybrid_clock.cc:648] HybridClock initialized: now 1781859296813655 us; error 53 us; skew 500 ppm
I20260619 08:54:56.814981 1318 webserver.cc:460] Webserver started at http://127.0.98.3:40477/ using document root <none> and password file <none>
I20260619 08:54:56.815220 1318 fs_manager.cc:359] Metadata directory not provided
I20260619 08:54:56.815280 1318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:54:56.815446 1318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:54:56.816380 1318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9"
format_stamp: "Formatted at 2026-06-19 08:54:56 on dist-test-slave-k50r"
I20260619 08:54:56.816773 1318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9"
format_stamp: "Formatted at 2026-06-19 08:54:56 on dist-test-slave-k50r"
I20260619 08:54:56.818126 1318 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 08:54:56.818938 1332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:56.819111 1318 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:56.819214 1318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9"
format_stamp: "Formatted at 2026-06-19 08:54:56 on dist-test-slave-k50r"
I20260619 08:54:56.819300 1318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:54:56.826184 1318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:54:56.826493 1318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:54:56.826639 1318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:54:56.826846 1318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:54:56.827167 1318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:54:56.827234 1318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:56.827271 1318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:54:56.827301 1318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:56.833420 1318 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45507
I20260619 08:54:56.833449 1445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45507 every 8 connection(s)
I20260619 08:54:56.833761 1318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20260619 08:54:56.839386 1446 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43691
I20260619 08:54:56.839465 1446 heartbeater.cc:464] Registering TS with master...
I20260619 08:54:56.839591 1446 heartbeater.cc:511] Master 127.0.98.62:43691 requested a full tablet report, sending...
I20260619 08:54:56.839926 996 ts_manager.cc:194] Registered new tserver with Master: 4e0e4dfff5e94d4b8e4b97882c68c5a9 (127.0.98.3:45507)
I20260619 08:54:56.840279 996 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:49281
I20260619 08:54:56.841048 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXnrr3A/build/release/bin/kudu as pid 1318
I20260619 08:54:56.841121 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20260619 08:54:56.842092 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXnrr3A/build/release/bin/kudu
/tmp/dist-test-taskXnrr3A/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.4:0
--local_ip_for_outbound_sockets=127.0.98.4
--webserver_interface=127.0.98.4
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43691
--builtin_ntp_servers=127.0.98.20:35761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
W20260619 08:54:56.921344 1449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:54:56.921588 1449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:54:56.921623 1449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:54:56.922973 1449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:54:56.923022 1449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.4
I20260619 08:54:56.924288 1449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.98.4
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43691
--never_fsync=true
--heap_profile_path=/tmp/kudu.1449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.4
--log_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:54:56.924450 1449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:54:56.924664 1449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:54:56.929302 1454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:54:56.929355 1455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:54:56.929358 1457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:54:56.929592 1449 server_base.cc:1034] running on GCE node
I20260619 08:54:56.929751 1449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:54:56.929921 1449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:54:56.931053 1449 hybrid_clock.cc:648] HybridClock initialized: now 1781859296931047 us; error 30 us; skew 500 ppm
I20260619 08:54:56.932194 1449 webserver.cc:460] Webserver started at http://127.0.98.4:34733/ using document root <none> and password file <none>
I20260619 08:54:56.932394 1449 fs_manager.cc:359] Metadata directory not provided
I20260619 08:54:56.932432 1449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:54:56.932516 1449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:54:56.933377 1449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "b6329eab306a4d6fa208557bd278a5d7"
format_stamp: "Formatted at 2026-06-19 08:54:56 on dist-test-slave-k50r"
I20260619 08:54:56.933656 1449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "b6329eab306a4d6fa208557bd278a5d7"
format_stamp: "Formatted at 2026-06-19 08:54:56 on dist-test-slave-k50r"
I20260619 08:54:56.934926 1449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:54:56.935564 1463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:56.935705 1449 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:54:56.935791 1449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "b6329eab306a4d6fa208557bd278a5d7"
format_stamp: "Formatted at 2026-06-19 08:54:56 on dist-test-slave-k50r"
I20260619 08:54:56.935839 1449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:54:56.953269 1449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:54:56.953466 1449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:54:56.953553 1449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:54:56.953691 1449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:54:56.953930 1449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:54:56.953958 1449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:56.953976 1449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:54:56.953989 1449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:56.959776 1449 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.4:35389
I20260619 08:54:56.959852 1576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.4:35389 every 8 connection(s)
I20260619 08:54:56.960147 1449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20260619 08:54:56.965793 1577 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43691
I20260619 08:54:56.965930 1577 heartbeater.cc:464] Registering TS with master...
I20260619 08:54:56.966111 1577 heartbeater.cc:511] Master 127.0.98.62:43691 requested a full tablet report, sending...
I20260619 08:54:56.966542 996 ts_manager.cc:194] Registered new tserver with Master: b6329eab306a4d6fa208557bd278a5d7 (127.0.98.4:35389)
I20260619 08:54:56.966668 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXnrr3A/build/release/bin/kudu as pid 1449
I20260619 08:54:56.966746 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20260619 08:54:56.966979 996 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.4:48819
I20260619 08:54:56.967653 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXnrr3A/build/release/bin/kudu
/tmp/dist-test-taskXnrr3A/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.5:0
--local_ip_for_outbound_sockets=127.0.98.5
--webserver_interface=127.0.98.5
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43691
--builtin_ntp_servers=127.0.98.20:35761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
W20260619 08:54:57.047562 1580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:54:57.047757 1580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:54:57.047776 1580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:54:57.048856 1580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:54:57.048898 1580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.5
I20260619 08:54:57.050163 1580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.98.5
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43691
--never_fsync=true
--heap_profile_path=/tmp/kudu.1580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.5
--log_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:54:57.050333 1580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:54:57.050613 1580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:54:57.055385 1585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:54:57.055508 1580 server_base.cc:1034] running on GCE node
W20260619 08:54:57.055397 1586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:54:57.055593 1588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:54:57.055837 1580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:54:57.056074 1580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:54:57.057225 1580 hybrid_clock.cc:648] HybridClock initialized: now 1781859297057210 us; error 33 us; skew 500 ppm
I20260619 08:54:57.058542 1580 webserver.cc:460] Webserver started at http://127.0.98.5:36759/ using document root <none> and password file <none>
I20260619 08:54:57.058792 1580 fs_manager.cc:359] Metadata directory not provided
I20260619 08:54:57.058841 1580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:54:57.058979 1580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:54:57.059911 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "171a74bcea154d9497fdbc1dff677801"
format_stamp: "Formatted at 2026-06-19 08:54:57 on dist-test-slave-k50r"
I20260619 08:54:57.060261 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "171a74bcea154d9497fdbc1dff677801"
format_stamp: "Formatted at 2026-06-19 08:54:57 on dist-test-slave-k50r"
I20260619 08:54:57.061510 1580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:54:57.062139 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:57.062319 1580 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:54:57.062415 1580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "171a74bcea154d9497fdbc1dff677801"
format_stamp: "Formatted at 2026-06-19 08:54:57 on dist-test-slave-k50r"
I20260619 08:54:57.062498 1580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:54:57.088477 1580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:54:57.088770 1580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:54:57.088954 1580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:54:57.089174 1580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:54:57.089500 1580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:54:57.089555 1580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:57.089594 1580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:54:57.089622 1580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:57.095105 1580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.5:38567
I20260619 08:54:57.095146 1707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.5:38567 every 8 connection(s)
I20260619 08:54:57.095498 1580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20260619 08:54:57.101347 1708 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43691
I20260619 08:54:57.101423 1708 heartbeater.cc:464] Registering TS with master...
I20260619 08:54:57.101590 1708 heartbeater.cc:511] Master 127.0.98.62:43691 requested a full tablet report, sending...
I20260619 08:54:57.102039 996 ts_manager.cc:194] Registered new tserver with Master: 171a74bcea154d9497fdbc1dff677801 (127.0.98.5:38567)
I20260619 08:54:57.102043 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXnrr3A/build/release/bin/kudu as pid 1580
I20260619 08:54:57.102142 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20260619 08:54:57.102522 996 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.5:39729
I20260619 08:54:57.103440 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXnrr3A/build/release/bin/kudu
/tmp/dist-test-taskXnrr3A/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.6:0
--local_ip_for_outbound_sockets=127.0.98.6
--webserver_interface=127.0.98.6
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43691
--builtin_ntp_servers=127.0.98.20:35761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
W20260619 08:54:57.183226 1711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:54:57.183455 1711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:54:57.183485 1711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:54:57.184589 1711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:54:57.184666 1711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.6
I20260619 08:54:57.185860 1711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.98.6
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43691
--never_fsync=true
--heap_profile_path=/tmp/kudu.1711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.6
--log_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:54:57.186122 1711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:54:57.186391 1711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:54:57.191148 1717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:54:57.191165 1716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:54:57.191202 1719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:54:57.191979 1711 server_base.cc:1034] running on GCE node
I20260619 08:54:57.192268 1711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:54:57.192520 1711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:54:57.193692 1711 hybrid_clock.cc:648] HybridClock initialized: now 1781859297193674 us; error 28 us; skew 500 ppm
I20260619 08:54:57.194969 1711 webserver.cc:460] Webserver started at http://127.0.98.6:38615/ using document root <none> and password file <none>
I20260619 08:54:57.195210 1711 fs_manager.cc:359] Metadata directory not provided
I20260619 08:54:57.195287 1711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:54:57.195420 1711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:54:57.196455 1711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "8380f1103150498b9eaebf58f4c49b7b"
format_stamp: "Formatted at 2026-06-19 08:54:57 on dist-test-slave-k50r"
I20260619 08:54:57.196822 1711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "8380f1103150498b9eaebf58f4c49b7b"
format_stamp: "Formatted at 2026-06-19 08:54:57 on dist-test-slave-k50r"
I20260619 08:54:57.198253 1711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 08:54:57.199054 1725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:57.199221 1711 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 08:54:57.199358 1711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "8380f1103150498b9eaebf58f4c49b7b"
format_stamp: "Formatted at 2026-06-19 08:54:57 on dist-test-slave-k50r"
I20260619 08:54:57.199431 1711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:54:57.214272 1711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:54:57.214609 1711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:54:57.214742 1711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:54:57.214942 1711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:54:57.215231 1711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:54:57.215289 1711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:57.215330 1711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:54:57.215358 1711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:57.221727 1711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.6:38855
I20260619 08:54:57.221843 1838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.6:38855 every 8 connection(s)
I20260619 08:54:57.222116 1711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20260619 08:54:57.227430 1839 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43691
I20260619 08:54:57.227524 1839 heartbeater.cc:464] Registering TS with master...
I20260619 08:54:57.227682 1839 heartbeater.cc:511] Master 127.0.98.62:43691 requested a full tablet report, sending...
I20260619 08:54:57.228056 995 ts_manager.cc:194] Registered new tserver with Master: 8380f1103150498b9eaebf58f4c49b7b (127.0.98.6:38855)
I20260619 08:54:57.228253 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXnrr3A/build/release/bin/kudu as pid 1711
I20260619 08:54:57.228324 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20260619 08:54:57.228538 995 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.6:41775
I20260619 08:54:57.229346 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskXnrr3A/build/release/bin/kudu
/tmp/dist-test-taskXnrr3A/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.7:0
--local_ip_for_outbound_sockets=127.0.98.7
--webserver_interface=127.0.98.7
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43691
--builtin_ntp_servers=127.0.98.20:35761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
W20260619 08:54:57.315501 1842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 08:54:57.315732 1842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 08:54:57.315750 1842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 08:54:57.316797 1842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 08:54:57.316838 1842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.7
I20260619 08:54:57.318068 1842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.98.7
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43691
--never_fsync=true
--heap_profile_path=/tmp/kudu.1842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.7
--log_dir=/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 08:54:57.318243 1842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 08:54:57.318481 1842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 08:54:57.323194 1847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:54:57.323261 1850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 08:54:57.323313 1848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 08:54:57.323590 1842 server_base.cc:1034] running on GCE node
I20260619 08:54:57.323800 1842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 08:54:57.324079 1842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 08:54:57.325289 1842 hybrid_clock.cc:648] HybridClock initialized: now 1781859297325283 us; error 37 us; skew 500 ppm
I20260619 08:54:57.326732 1842 webserver.cc:460] Webserver started at http://127.0.98.7:35713/ using document root <none> and password file <none>
I20260619 08:54:57.326962 1842 fs_manager.cc:359] Metadata directory not provided
I20260619 08:54:57.327006 1842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 08:54:57.327168 1842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 08:54:57.328116 1842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "aa6ea950b6234668969492322c29714b"
format_stamp: "Formatted at 2026-06-19 08:54:57 on dist-test-slave-k50r"
I20260619 08:54:57.328476 1842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "aa6ea950b6234668969492322c29714b"
format_stamp: "Formatted at 2026-06-19 08:54:57 on dist-test-slave-k50r"
I20260619 08:54:57.329993 1842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 08:54:57.330899 1856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:57.331067 1842 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:57.331140 1842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "aa6ea950b6234668969492322c29714b"
format_stamp: "Formatted at 2026-06-19 08:54:57 on dist-test-slave-k50r"
I20260619 08:54:57.331190 1842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 08:54:57.352604 1842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 08:54:57.352849 1842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 08:54:57.352977 1842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 08:54:57.353222 1842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 08:54:57.353543 1842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 08:54:57.353576 1842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:57.353598 1842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 08:54:57.353610 1842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 08:54:57.360359 1842 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.7:41045
I20260619 08:54:57.360441 1969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.7:41045 every 8 connection(s)
I20260619 08:54:57.360765 1842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20260619 08:54:57.363960 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskXnrr3A/build/release/bin/kudu as pid 1842
I20260619 08:54:57.364054 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskXnrr3A/test-tmp/raft_consensus-itest.2.RaftConsensusITest.MultiThreadedInsertWithFailovers.1781859294830291-392-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20260619 08:54:57.366894 1970 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43691
I20260619 08:54:57.366994 1970 heartbeater.cc:464] Registering TS with master...
I20260619 08:54:57.367161 1970 heartbeater.cc:511] Master 127.0.98.62:43691 requested a full tablet report, sending...
I20260619 08:54:57.367484 995 ts_manager.cc:194] Registered new tserver with Master: aa6ea950b6234668969492322c29714b (127.0.98.7:41045)
I20260619 08:54:57.367848 995 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.7:56911
I20260619 08:54:57.376565 392 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20260619 08:54:57.382786 995 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50626:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20260619 08:54:57.385773 995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20260619 08:54:57.393658 1642 tablet_service.cc:1467] Processing CreateTablet for tablet 63f80b9eea6c49659b4407c51e019b96 (DEFAULT_TABLE table=TestTable [id=744f75b33ed24b348e88e478122b4aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:54:57.393666 1773 tablet_service.cc:1467] Processing CreateTablet for tablet 63f80b9eea6c49659b4407c51e019b96 (DEFAULT_TABLE table=TestTable [id=744f75b33ed24b348e88e478122b4aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:54:57.393980 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63f80b9eea6c49659b4407c51e019b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:54:57.394060 1773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63f80b9eea6c49659b4407c51e019b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:54:57.396792 1904 tablet_service.cc:1467] Processing CreateTablet for tablet 63f80b9eea6c49659b4407c51e019b96 (DEFAULT_TABLE table=TestTable [id=744f75b33ed24b348e88e478122b4aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:54:57.397092 1904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63f80b9eea6c49659b4407c51e019b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:54:57.397090 1249 tablet_service.cc:1467] Processing CreateTablet for tablet 63f80b9eea6c49659b4407c51e019b96 (DEFAULT_TABLE table=TestTable [id=744f75b33ed24b348e88e478122b4aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:54:57.397372 1995 tablet_bootstrap.cc:492] T 63f80b9eea6c49659b4407c51e019b96 P 171a74bcea154d9497fdbc1dff677801: Bootstrap starting.
I20260619 08:54:57.397401 1249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63f80b9eea6c49659b4407c51e019b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:54:57.397995 1995 tablet_bootstrap.cc:654] T 63f80b9eea6c49659b4407c51e019b96 P 171a74bcea154d9497fdbc1dff677801: Neither blocks nor log segments found. Creating new log.
I20260619 08:54:57.398290 1995 log.cc:824] T 63f80b9eea6c49659b4407c51e019b96 P 171a74bcea154d9497fdbc1dff677801: Log is configured to *not* fsync() on all Append() calls
I20260619 08:54:57.399549 1380 tablet_service.cc:1467] Processing CreateTablet for tablet 63f80b9eea6c49659b4407c51e019b96 (DEFAULT_TABLE table=TestTable [id=744f75b33ed24b348e88e478122b4aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:54:57.399852 1380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63f80b9eea6c49659b4407c51e019b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:54:57.401865 1118 tablet_service.cc:1467] Processing CreateTablet for tablet 63f80b9eea6c49659b4407c51e019b96 (DEFAULT_TABLE table=TestTable [id=744f75b33ed24b348e88e478122b4aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:54:57.402199 1118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63f80b9eea6c49659b4407c51e019b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:54:57.402992 2000 tablet_bootstrap.cc:492] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9: Bootstrap starting.
I20260619 08:54:57.403277 1995 tablet_bootstrap.cc:492] T 63f80b9eea6c49659b4407c51e019b96 P 171a74bcea154d9497fdbc1dff677801: No bootstrap required, opened a new log
I20260619 08:54:57.403349 1995 ts_tablet_manager.cc:1397] T 63f80b9eea6c49659b4407c51e019b96 P 171a74bcea154d9497fdbc1dff677801: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260619 08:54:57.403808 2000 tablet_bootstrap.cc:654] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9: Neither blocks nor log segments found. Creating new log.
I20260619 08:54:57.404165 1997 tablet_bootstrap.cc:492] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b: Bootstrap starting.
I20260619 08:54:57.404309 2000 log.cc:824] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9: Log is configured to *not* fsync() on all Append() calls
I20260619 08:54:57.404884 1997 tablet_bootstrap.cc:654] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b: Neither blocks nor log segments found. Creating new log.
I20260619 08:54:57.405165 1997 log.cc:824] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b: Log is configured to *not* fsync() on all Append() calls
I20260619 08:54:57.405277 1511 tablet_service.cc:1467] Processing CreateTablet for tablet 63f80b9eea6c49659b4407c51e019b96 (DEFAULT_TABLE table=TestTable [id=744f75b33ed24b348e88e478122b4aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 08:54:57.405486 1998 tablet_bootstrap.cc:492] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b: Bootstrap starting.
I20260619 08:54:57.405592 1511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63f80b9eea6c49659b4407c51e019b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 08:54:57.406318 2000 tablet_bootstrap.cc:492] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9: No bootstrap required, opened a new log
I20260619 08:54:57.406373 2000 ts_tablet_manager.cc:1397] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 08:54:57.406550 1998 tablet_bootstrap.cc:654] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b: Neither blocks nor log segments found. Creating new log.
I20260619 08:54:57.406629 1995 raft_consensus.cc:348] T 63f80b9eea6c49659b4407c51e019b96 P 171a74bcea154d9497fdbc1dff677801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:54:57.406823 1995 raft_consensus.cc:374] T 63f80b9eea6c49659b4407c51e019b96 P 171a74bcea154d9497fdbc1dff677801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:54:57.406823 1998 log.cc:824] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b: Log is configured to *not* fsync() on all Append() calls
I20260619 08:54:57.406857 1995 raft_consensus.cc:729] T 63f80b9eea6c49659b4407c51e019b96 P 171a74bcea154d9497fdbc1dff677801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171a74bcea154d9497fdbc1dff677801, State: Initialized, Role: FOLLOWER
I20260619 08:54:57.406991 1995 consensus_queue.cc:260] T 63f80b9eea6c49659b4407c51e019b96 P 171a74bcea154d9497fdbc1dff677801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:54:57.407106 1999 tablet_bootstrap.cc:492] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01: Bootstrap starting.
I20260619 08:54:57.407253 1995 ts_tablet_manager.cc:1428] T 63f80b9eea6c49659b4407c51e019b96 P 171a74bcea154d9497fdbc1dff677801: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260619 08:54:57.407359 1708 heartbeater.cc:503] Master 127.0.98.62:43691 was elected leader, sending a full tablet report...
I20260619 08:54:57.407697 1999 tablet_bootstrap.cc:654] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01: Neither blocks nor log segments found. Creating new log.
I20260619 08:54:57.407958 1999 log.cc:824] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01: Log is configured to *not* fsync() on all Append() calls
I20260619 08:54:57.408603 1997 tablet_bootstrap.cc:492] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b: No bootstrap required, opened a new log
I20260619 08:54:57.408610 1999 tablet_bootstrap.cc:492] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01: No bootstrap required, opened a new log
I20260619 08:54:57.408659 1999 ts_tablet_manager.cc:1397] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:54:57.408659 1997 ts_tablet_manager.cc:1397] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260619 08:54:57.409232 2000 raft_consensus.cc:348] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:54:57.409392 2000 raft_consensus.cc:374] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:54:57.409430 2000 raft_consensus.cc:729] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e0e4dfff5e94d4b8e4b97882c68c5a9, State: Initialized, Role: FOLLOWER
I20260619 08:54:57.409603 2000 consensus_queue.cc:260] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:54:57.409897 2000 ts_tablet_manager.cc:1428] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 08:54:57.410004 1446 heartbeater.cc:503] Master 127.0.98.62:43691 was elected leader, sending a full tablet report...
I20260619 08:54:57.411602 1999 raft_consensus.cc:348] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:54:57.411620 1997 raft_consensus.cc:348] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:54:57.411773 1997 raft_consensus.cc:374] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:54:57.411774 1999 raft_consensus.cc:374] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:54:57.411809 1999 raft_consensus.cc:729] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 408d1f4ba9f245f1aa2608445e067a01, State: Initialized, Role: FOLLOWER
I20260619 08:54:57.411809 1997 raft_consensus.cc:729] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa6ea950b6234668969492322c29714b, State: Initialized, Role: FOLLOWER
I20260619 08:54:57.411933 1999 consensus_queue.cc:260] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:54:57.412051 1997 consensus_queue.cc:260] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:54:57.412221 1999 ts_tablet_manager.cc:1428] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260619 08:54:57.412334 1315 heartbeater.cc:503] Master 127.0.98.62:43691 was elected leader, sending a full tablet report...
I20260619 08:54:57.412294 1997 ts_tablet_manager.cc:1428] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 08:54:57.412350 1970 heartbeater.cc:503] Master 127.0.98.62:43691 was elected leader, sending a full tablet report...
I20260619 08:54:57.412783 2003 tablet_bootstrap.cc:492] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb: Bootstrap starting.
I20260619 08:54:57.413429 2003 tablet_bootstrap.cc:654] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb: Neither blocks nor log segments found. Creating new log.
I20260619 08:54:57.413692 2003 log.cc:824] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb: Log is configured to *not* fsync() on all Append() calls
I20260619 08:54:57.414173 2007 tablet_bootstrap.cc:492] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7: Bootstrap starting.
I20260619 08:54:57.414374 2003 tablet_bootstrap.cc:492] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb: No bootstrap required, opened a new log
I20260619 08:54:57.414388 1998 tablet_bootstrap.cc:492] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b: No bootstrap required, opened a new log
I20260619 08:54:57.414469 2003 ts_tablet_manager.cc:1397] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:54:57.414484 1998 ts_tablet_manager.cc:1397] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20260619 08:54:57.414806 2007 tablet_bootstrap.cc:654] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7: Neither blocks nor log segments found. Creating new log.
I20260619 08:54:57.415098 2007 log.cc:824] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7: Log is configured to *not* fsync() on all Append() calls
I20260619 08:54:57.415858 2007 tablet_bootstrap.cc:492] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7: No bootstrap required, opened a new log
I20260619 08:54:57.415926 2007 ts_tablet_manager.cc:1397] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 08:54:57.416868 2003 raft_consensus.cc:348] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:54:57.417016 2003 raft_consensus.cc:374] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:54:57.417057 2003 raft_consensus.cc:729] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f2ab318c954848b2be15d0926b94fb, State: Initialized, Role: FOLLOWER
I20260619 08:54:57.417052 1998 raft_consensus.cc:348] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:54:57.417240 1998 raft_consensus.cc:374] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:54:57.417186 2003 consensus_queue.cc:260] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:54:57.417284 1998 raft_consensus.cc:729] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8380f1103150498b9eaebf58f4c49b7b, State: Initialized, Role: FOLLOWER
I20260619 08:54:57.417402 1998 consensus_queue.cc:260] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:54:57.417665 1998 ts_tablet_manager.cc:1428] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 08:54:57.417776 1839 heartbeater.cc:503] Master 127.0.98.62:43691 was elected leader, sending a full tablet report...
I20260619 08:54:57.417896 2003 ts_tablet_manager.cc:1428] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 08:54:57.417985 1184 heartbeater.cc:503] Master 127.0.98.62:43691 was elected leader, sending a full tablet report...
I20260619 08:54:57.418767 2007 raft_consensus.cc:348] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:54:57.418887 2007 raft_consensus.cc:374] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 08:54:57.418907 2007 raft_consensus.cc:729] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6329eab306a4d6fa208557bd278a5d7, State: Initialized, Role: FOLLOWER
I20260619 08:54:57.419011 2007 consensus_queue.cc:260] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:54:57.419289 2007 ts_tablet_manager.cc:1428] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 08:54:57.419301 1577 heartbeater.cc:503] Master 127.0.98.62:43691 was elected leader, sending a full tablet report...
W20260619 08:54:57.460783 1578 tablet.cc:2369] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:54:57.470288 1316 tablet.cc:2369] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:54:57.472649 2015 raft_consensus.cc:482] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:54:57.472738 2015 raft_consensus.cc:504] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:54:57.473634 2015 leader_election.cc:288] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83f2ab318c954848b2be15d0926b94fb (127.0.98.1:39977), 8380f1103150498b9eaebf58f4c49b7b (127.0.98.6:38855), aa6ea950b6234668969492322c29714b (127.0.98.7:41045), 171a74bcea154d9497fdbc1dff677801 (127.0.98.5:38567), 408d1f4ba9f245f1aa2608445e067a01 (127.0.98.2:46549), 4e0e4dfff5e94d4b8e4b97882c68c5a9 (127.0.98.3:45507)
W20260619 08:54:57.474535 1840 tablet.cc:2369] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:54:57.477805 1793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "b6329eab306a4d6fa208557bd278a5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8380f1103150498b9eaebf58f4c49b7b" is_pre_election: true
I20260619 08:54:57.477953 1793 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6329eab306a4d6fa208557bd278a5d7 in term 0.
I20260619 08:54:57.478140 1269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "b6329eab306a4d6fa208557bd278a5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "408d1f4ba9f245f1aa2608445e067a01" is_pre_election: true
I20260619 08:54:57.478281 1269 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6329eab306a4d6fa208557bd278a5d7 in term 0.
I20260619 08:54:57.478725 1924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "b6329eab306a4d6fa208557bd278a5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa6ea950b6234668969492322c29714b" is_pre_election: true
I20260619 08:54:57.478871 1924 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6329eab306a4d6fa208557bd278a5d7 in term 0.
I20260619 08:54:57.479003 1138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "b6329eab306a4d6fa208557bd278a5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f2ab318c954848b2be15d0926b94fb" is_pre_election: true
I20260619 08:54:57.479125 1138 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6329eab306a4d6fa208557bd278a5d7 in term 0.
I20260619 08:54:57.479202 1467 leader_election.cc:302] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 408d1f4ba9f245f1aa2608445e067a01, 8380f1103150498b9eaebf58f4c49b7b, aa6ea950b6234668969492322c29714b, b6329eab306a4d6fa208557bd278a5d7; no voters:
I20260619 08:54:57.479303 2015 raft_consensus.cc:2781] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 08:54:57.479351 2015 raft_consensus.cc:482] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:54:57.479387 2015 raft_consensus.cc:3037] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:54:57.481509 1400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "b6329eab306a4d6fa208557bd278a5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" is_pre_election: true
I20260619 08:54:57.481673 1400 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6329eab306a4d6fa208557bd278a5d7 in term 0.
I20260619 08:54:57.481791 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "b6329eab306a4d6fa208557bd278a5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171a74bcea154d9497fdbc1dff677801" is_pre_election: true
I20260619 08:54:57.481963 1662 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 171a74bcea154d9497fdbc1dff677801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6329eab306a4d6fa208557bd278a5d7 in term 0.
I20260619 08:54:57.482996 2015 raft_consensus.cc:504] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:54:57.483319 1138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "b6329eab306a4d6fa208557bd278a5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f2ab318c954848b2be15d0926b94fb"
I20260619 08:54:57.483335 1924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "b6329eab306a4d6fa208557bd278a5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa6ea950b6234668969492322c29714b"
I20260619 08:54:57.483399 1269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "b6329eab306a4d6fa208557bd278a5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "408d1f4ba9f245f1aa2608445e067a01"
I20260619 08:54:57.483410 1924 raft_consensus.cc:3037] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:54:57.483404 1138 raft_consensus.cc:3037] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:54:57.483461 1269 raft_consensus.cc:3037] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:54:57.484226 1269 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6329eab306a4d6fa208557bd278a5d7 in term 1.
I20260619 08:54:57.484226 1924 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6329eab306a4d6fa208557bd278a5d7 in term 1.
I20260619 08:54:57.484385 1138 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6329eab306a4d6fa208557bd278a5d7 in term 1.
I20260619 08:54:57.484463 2015 leader_election.cc:288] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [CANDIDATE]: Term 1 election: Requested vote from peers 83f2ab318c954848b2be15d0926b94fb (127.0.98.1:39977), 8380f1103150498b9eaebf58f4c49b7b (127.0.98.6:38855), aa6ea950b6234668969492322c29714b (127.0.98.7:41045), 171a74bcea154d9497fdbc1dff677801 (127.0.98.5:38567), 408d1f4ba9f245f1aa2608445e067a01 (127.0.98.2:46549), 4e0e4dfff5e94d4b8e4b97882c68c5a9 (127.0.98.3:45507)
I20260619 08:54:57.484553 1464 leader_election.cc:302] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 408d1f4ba9f245f1aa2608445e067a01, 83f2ab318c954848b2be15d0926b94fb, aa6ea950b6234668969492322c29714b, b6329eab306a4d6fa208557bd278a5d7; no voters:
I20260619 08:54:57.484597 1400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "b6329eab306a4d6fa208557bd278a5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9"
I20260619 08:54:57.484648 2015 raft_consensus.cc:2781] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 08:54:57.484664 1400 raft_consensus.cc:3037] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:54:57.484788 2015 raft_consensus.cc:686] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [term 1 LEADER]: Becoming Leader. State: Replica: b6329eab306a4d6fa208557bd278a5d7, State: Running, Role: LEADER
I20260619 08:54:57.484880 2015 consensus_queue.cc:237] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:54:57.485169 1793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "b6329eab306a4d6fa208557bd278a5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8380f1103150498b9eaebf58f4c49b7b"
I20260619 08:54:57.485258 1793 raft_consensus.cc:3037] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:54:57.485512 1400 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6329eab306a4d6fa208557bd278a5d7 in term 1.
I20260619 08:54:57.486006 995 catalog_manager.cc:5495] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6329eab306a4d6fa208557bd278a5d7 (127.0.98.4). New cstate: current_term: 1 leader_uuid: "b6329eab306a4d6fa208557bd278a5d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } health_report { overall_health: UNKNOWN } } }
I20260619 08:54:57.486191 1793 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6329eab306a4d6fa208557bd278a5d7 in term 1.
I20260619 08:54:57.486481 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "b6329eab306a4d6fa208557bd278a5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171a74bcea154d9497fdbc1dff677801"
I20260619 08:54:57.486572 1662 raft_consensus.cc:3037] T 63f80b9eea6c49659b4407c51e019b96 P 171a74bcea154d9497fdbc1dff677801 [term 0 FOLLOWER]: Advancing to term 1
I20260619 08:54:57.489935 392 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20260619 08:54:57.490494 392 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 83f2ab318c954848b2be15d0926b94fb to finish bootstrapping
I20260619 08:54:57.494325 392 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 408d1f4ba9f245f1aa2608445e067a01 to finish bootstrapping
I20260619 08:54:57.497643 392 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4e0e4dfff5e94d4b8e4b97882c68c5a9 to finish bootstrapping
I20260619 08:54:57.500840 392 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b6329eab306a4d6fa208557bd278a5d7 to finish bootstrapping
I20260619 08:54:57.504154 392 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 171a74bcea154d9497fdbc1dff677801 to finish bootstrapping
I20260619 08:54:57.575642 1662 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 171a74bcea154d9497fdbc1dff677801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6329eab306a4d6fa208557bd278a5d7 in term 1.
I20260619 08:54:57.576419 392 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 8380f1103150498b9eaebf58f4c49b7b to finish bootstrapping
I20260619 08:54:57.580477 392 ts_itest-base.cc:233] Waiting for 1 tablets on tserver aa6ea950b6234668969492322c29714b to finish bootstrapping
W20260619 08:54:57.584627 1447 tablet.cc:2369] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 08:54:57.591382 1185 tablet.cc:2369] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:54:57.591382 1400 raft_consensus.cc:1261] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9 [term 1 FOLLOWER]: Refusing update from remote peer b6329eab306a4d6fa208557bd278a5d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:54:57.591625 1269 raft_consensus.cc:1261] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01 [term 1 FOLLOWER]: Refusing update from remote peer b6329eab306a4d6fa208557bd278a5d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:54:57.591720 2022 consensus_queue.cc:1035] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:54:57.591943 1662 raft_consensus.cc:1261] T 63f80b9eea6c49659b4407c51e019b96 P 171a74bcea154d9497fdbc1dff677801 [term 1 FOLLOWER]: Refusing update from remote peer b6329eab306a4d6fa208557bd278a5d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:54:57.592207 1793 raft_consensus.cc:1261] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 1 FOLLOWER]: Refusing update from remote peer b6329eab306a4d6fa208557bd278a5d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:54:57.592478 1138 raft_consensus.cc:1261] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [term 1 FOLLOWER]: Refusing update from remote peer b6329eab306a4d6fa208557bd278a5d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:54:57.592777 2015 consensus_queue.cc:1035] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:54:57.592898 2015 consensus_queue.cc:1035] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:54:57.592959 2015 consensus_queue.cc:1035] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:54:57.593030 2015 consensus_queue.cc:1035] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 08:54:57.596333 1709 tablet.cc:2369] T 63f80b9eea6c49659b4407c51e019b96 P 171a74bcea154d9497fdbc1dff677801: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 08:54:57.596989 1924 raft_consensus.cc:1261] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [term 1 FOLLOWER]: Refusing update from remote peer b6329eab306a4d6fa208557bd278a5d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 08:54:57.597322 2015 consensus_queue.cc:1035] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 08:54:57.599431 2034 mvcc.cc:205] Tried to move back new op lower bound from 7298495682933424128 to 7298495682499153920. Current Snapshot: MvccSnapshot[applied={T|T < 7298495682930917376}]
I20260619 08:54:57.600001 2044 mvcc.cc:205] Tried to move back new op lower bound from 7298495682933424128 to 7298495682499153920. Current Snapshot: MvccSnapshot[applied={T|T < 7298495682930917376}]
I20260619 08:54:57.601003 2041 mvcc.cc:205] Tried to move back new op lower bound from 7298495682933424128 to 7298495682499153920. Current Snapshot: MvccSnapshot[applied={T|T < 7298495682930917376}]
I20260619 08:54:57.601475 2040 mvcc.cc:205] Tried to move back new op lower bound from 7298495682933424128 to 7298495682499153920. Current Snapshot: MvccSnapshot[applied={T|T < 7298495682933424128 or (T in {7298495682933424128})}]
I20260619 08:54:57.602160 2046 mvcc.cc:205] Tried to move back new op lower bound from 7298495682957852672 to 7298495682499153920. Current Snapshot: MvccSnapshot[applied={T|T < 7298495682930917376}]
I20260619 08:54:57.602600 2048 mvcc.cc:205] Tried to move back new op lower bound from 7298495682957852672 to 7298495682499153920. Current Snapshot: MvccSnapshot[applied={T|T < 7298495682930917376}]
I20260619 08:54:57.609181 2038 mvcc.cc:205] Tried to move back new op lower bound from 7298495682933424128 to 7298495682499153920. Current Snapshot: MvccSnapshot[applied={T|T < 7298495682933424128 or (T in {7298495682933424128})}]
I20260619 08:54:58.550550 392 test_util.cc:268] Using random seed: 1921491726
I20260619 08:54:58.870584 392 test_util.cc:268] Using random seed: 1921811756
I20260619 08:54:59.260521 2091 raft_consensus.cc:482] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b6329eab306a4d6fa208557bd278a5d7)
I20260619 08:54:59.260617 2091 raft_consensus.cc:504] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:54:59.261035 2091 leader_election.cc:288] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f2ab318c954848b2be15d0926b94fb (127.0.98.1:39977), 8380f1103150498b9eaebf58f4c49b7b (127.0.98.6:38855), aa6ea950b6234668969492322c29714b (127.0.98.7:41045), 171a74bcea154d9497fdbc1dff677801 (127.0.98.5:38567), 408d1f4ba9f245f1aa2608445e067a01 (127.0.98.2:46549), b6329eab306a4d6fa208557bd278a5d7 (127.0.98.4:35389)
I20260619 08:54:59.265617 1924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 236 } } ignore_live_leader: false dest_uuid: "aa6ea950b6234668969492322c29714b" is_pre_election: true
I20260619 08:54:59.265825 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 236 } } ignore_live_leader: false dest_uuid: "171a74bcea154d9497fdbc1dff677801" is_pre_election: true
I20260619 08:54:59.266587 1137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 236 } } ignore_live_leader: false dest_uuid: "83f2ab318c954848b2be15d0926b94fb" is_pre_election: true
I20260619 08:54:59.266783 1269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 236 } } ignore_live_leader: false dest_uuid: "408d1f4ba9f245f1aa2608445e067a01" is_pre_election: true
I20260619 08:54:59.267005 1335 leader_election.cc:302] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 4e0e4dfff5e94d4b8e4b97882c68c5a9; no voters: 171a74bcea154d9497fdbc1dff677801, 408d1f4ba9f245f1aa2608445e067a01, 83f2ab318c954848b2be15d0926b94fb, aa6ea950b6234668969492322c29714b
I20260619 08:54:59.267148 2091 raft_consensus.cc:2726] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 08:54:59.267323 1793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 236 } } ignore_live_leader: false dest_uuid: "8380f1103150498b9eaebf58f4c49b7b" is_pre_election: true
I20260619 08:54:59.275115 392 test_util.cc:268] Using random seed: 1922216291
I20260619 08:54:59.370708 2103 raft_consensus.cc:482] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b6329eab306a4d6fa208557bd278a5d7)
I20260619 08:54:59.370847 2103 raft_consensus.cc:504] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:54:59.371317 2103 leader_election.cc:288] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f2ab318c954848b2be15d0926b94fb (127.0.98.1:39977), 8380f1103150498b9eaebf58f4c49b7b (127.0.98.6:38855), aa6ea950b6234668969492322c29714b (127.0.98.7:41045), 171a74bcea154d9497fdbc1dff677801 (127.0.98.5:38567), b6329eab306a4d6fa208557bd278a5d7 (127.0.98.4:35389), 4e0e4dfff5e94d4b8e4b97882c68c5a9 (127.0.98.3:45507)
I20260619 08:54:59.375679 1137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "408d1f4ba9f245f1aa2608445e067a01" candidate_term: 2 candidate_status { last_received { term: 1 index: 421 } } ignore_live_leader: false dest_uuid: "83f2ab318c954848b2be15d0926b94fb" is_pre_election: true
I20260619 08:54:59.375984 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "408d1f4ba9f245f1aa2608445e067a01" candidate_term: 2 candidate_status { last_received { term: 1 index: 421 } } ignore_live_leader: false dest_uuid: "171a74bcea154d9497fdbc1dff677801" is_pre_election: true
I20260619 08:54:59.376667 1793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "408d1f4ba9f245f1aa2608445e067a01" candidate_term: 2 candidate_status { last_received { term: 1 index: 421 } } ignore_live_leader: false dest_uuid: "8380f1103150498b9eaebf58f4c49b7b" is_pre_election: true
I20260619 08:54:59.377312 1924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "408d1f4ba9f245f1aa2608445e067a01" candidate_term: 2 candidate_status { last_received { term: 1 index: 421 } } ignore_live_leader: false dest_uuid: "aa6ea950b6234668969492322c29714b" is_pre_election: true
I20260619 08:54:59.377532 1205 leader_election.cc:302] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 408d1f4ba9f245f1aa2608445e067a01; no voters: 171a74bcea154d9497fdbc1dff677801, 8380f1103150498b9eaebf58f4c49b7b, 83f2ab318c954848b2be15d0926b94fb, aa6ea950b6234668969492322c29714b
I20260619 08:54:59.377689 2103 raft_consensus.cc:2726] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 08:54:59.377998 1398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "408d1f4ba9f245f1aa2608445e067a01" candidate_term: 2 candidate_status { last_received { term: 1 index: 421 } } ignore_live_leader: false dest_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" is_pre_election: true
I20260619 08:54:59.378108 1398 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 408d1f4ba9f245f1aa2608445e067a01 in term 1.
I20260619 08:54:59.599674 2111 raft_consensus.cc:482] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b6329eab306a4d6fa208557bd278a5d7)
I20260619 08:54:59.599759 2111 raft_consensus.cc:504] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:54:59.600153 2112 raft_consensus.cc:482] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b6329eab306a4d6fa208557bd278a5d7)
I20260619 08:54:59.600296 2111 leader_election.cc:288] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8380f1103150498b9eaebf58f4c49b7b (127.0.98.6:38855), aa6ea950b6234668969492322c29714b (127.0.98.7:41045), 171a74bcea154d9497fdbc1dff677801 (127.0.98.5:38567), 408d1f4ba9f245f1aa2608445e067a01 (127.0.98.2:46549), b6329eab306a4d6fa208557bd278a5d7 (127.0.98.4:35389), 4e0e4dfff5e94d4b8e4b97882c68c5a9 (127.0.98.3:45507)
I20260619 08:54:59.600553 2112 raft_consensus.cc:504] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:54:59.601750 2112 leader_election.cc:288] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f2ab318c954848b2be15d0926b94fb (127.0.98.1:39977), aa6ea950b6234668969492322c29714b (127.0.98.7:41045), 171a74bcea154d9497fdbc1dff677801 (127.0.98.5:38567), 408d1f4ba9f245f1aa2608445e067a01 (127.0.98.2:46549), b6329eab306a4d6fa208557bd278a5d7 (127.0.98.4:35389), 4e0e4dfff5e94d4b8e4b97882c68c5a9 (127.0.98.3:45507)
I20260619 08:54:59.605633 1924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "83f2ab318c954848b2be15d0926b94fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 863 } } ignore_live_leader: false dest_uuid: "aa6ea950b6234668969492322c29714b" is_pre_election: true
I20260619 08:54:59.608809 1793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "83f2ab318c954848b2be15d0926b94fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 863 } } ignore_live_leader: false dest_uuid: "8380f1103150498b9eaebf58f4c49b7b" is_pre_election: true
I20260619 08:54:59.608925 1793 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f2ab318c954848b2be15d0926b94fb in term 1.
I20260619 08:54:59.609287 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "83f2ab318c954848b2be15d0926b94fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 863 } } ignore_live_leader: false dest_uuid: "171a74bcea154d9497fdbc1dff677801" is_pre_election: true
I20260619 08:54:59.609413 1137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "8380f1103150498b9eaebf58f4c49b7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 863 } } ignore_live_leader: false dest_uuid: "83f2ab318c954848b2be15d0926b94fb" is_pre_election: true
I20260619 08:54:59.609488 1137 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8380f1103150498b9eaebf58f4c49b7b in term 1.
I20260619 08:54:59.610178 1398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "83f2ab318c954848b2be15d0926b94fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 863 } } ignore_live_leader: false dest_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" is_pre_election: true
I20260619 08:54:59.610280 1398 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f2ab318c954848b2be15d0926b94fb in term 1.
I20260619 08:54:59.610484 1269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "83f2ab318c954848b2be15d0926b94fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 863 } } ignore_live_leader: false dest_uuid: "408d1f4ba9f245f1aa2608445e067a01" is_pre_election: true
I20260619 08:54:59.610582 1269 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f2ab318c954848b2be15d0926b94fb in term 1.
I20260619 08:54:59.610730 1073 leader_election.cc:302] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 408d1f4ba9f245f1aa2608445e067a01, 4e0e4dfff5e94d4b8e4b97882c68c5a9, 8380f1103150498b9eaebf58f4c49b7b, 83f2ab318c954848b2be15d0926b94fb; no voters: 171a74bcea154d9497fdbc1dff677801, aa6ea950b6234668969492322c29714b
I20260619 08:54:59.610857 2111 raft_consensus.cc:2781] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 08:54:59.610898 2111 raft_consensus.cc:482] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [term 1 FOLLOWER]: Starting leader election (detected failure of leader b6329eab306a4d6fa208557bd278a5d7)
I20260619 08:54:59.610922 2111 raft_consensus.cc:3037] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [term 1 FOLLOWER]: Advancing to term 2
I20260619 08:54:59.611552 1924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "8380f1103150498b9eaebf58f4c49b7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 863 } } ignore_live_leader: false dest_uuid: "aa6ea950b6234668969492322c29714b" is_pre_election: true
I20260619 08:54:59.612788 1269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "8380f1103150498b9eaebf58f4c49b7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 863 } } ignore_live_leader: false dest_uuid: "408d1f4ba9f245f1aa2608445e067a01" is_pre_election: true
I20260619 08:54:59.612883 1269 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8380f1103150498b9eaebf58f4c49b7b in term 1.
I20260619 08:54:59.613286 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "8380f1103150498b9eaebf58f4c49b7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 863 } } ignore_live_leader: false dest_uuid: "171a74bcea154d9497fdbc1dff677801" is_pre_election: true
I20260619 08:54:59.614125 1398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "8380f1103150498b9eaebf58f4c49b7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 863 } } ignore_live_leader: false dest_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" is_pre_election: true
I20260619 08:54:59.614198 1398 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8380f1103150498b9eaebf58f4c49b7b in term 1.
I20260619 08:54:59.614362 1728 leader_election.cc:302] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 408d1f4ba9f245f1aa2608445e067a01, 4e0e4dfff5e94d4b8e4b97882c68c5a9, 8380f1103150498b9eaebf58f4c49b7b, 83f2ab318c954848b2be15d0926b94fb; no voters: 171a74bcea154d9497fdbc1dff677801, aa6ea950b6234668969492322c29714b
I20260619 08:54:59.614532 2112 raft_consensus.cc:2781] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 08:54:59.614598 2112 raft_consensus.cc:482] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 1 FOLLOWER]: Starting leader election (detected failure of leader b6329eab306a4d6fa208557bd278a5d7)
I20260619 08:54:59.614626 2112 raft_consensus.cc:3037] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 1 FOLLOWER]: Advancing to term 2
I20260619 08:54:59.700100 2111 raft_consensus.cc:504] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:54:59.700101 2112 raft_consensus.cc:504] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:54:59.700347 2111 leader_election.cc:288] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [CANDIDATE]: Term 2 election: Requested vote from peers 8380f1103150498b9eaebf58f4c49b7b (127.0.98.6:38855), aa6ea950b6234668969492322c29714b (127.0.98.7:41045), 171a74bcea154d9497fdbc1dff677801 (127.0.98.5:38567), 408d1f4ba9f245f1aa2608445e067a01 (127.0.98.2:46549), b6329eab306a4d6fa208557bd278a5d7 (127.0.98.4:35389), 4e0e4dfff5e94d4b8e4b97882c68c5a9 (127.0.98.3:45507)
I20260619 08:54:59.700373 2112 leader_election.cc:288] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [CANDIDATE]: Term 2 election: Requested vote from peers 83f2ab318c954848b2be15d0926b94fb (127.0.98.1:39977), aa6ea950b6234668969492322c29714b (127.0.98.7:41045), 171a74bcea154d9497fdbc1dff677801 (127.0.98.5:38567), 408d1f4ba9f245f1aa2608445e067a01 (127.0.98.2:46549), b6329eab306a4d6fa208557bd278a5d7 (127.0.98.4:35389), 4e0e4dfff5e94d4b8e4b97882c68c5a9 (127.0.98.3:45507)
I20260619 08:54:59.700579 1137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "8380f1103150498b9eaebf58f4c49b7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 863 } } ignore_live_leader: false dest_uuid: "83f2ab318c954848b2be15d0926b94fb"
I20260619 08:54:59.700671 1137 raft_consensus.cc:2379] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8380f1103150498b9eaebf58f4c49b7b in current term 2: Already voted for candidate 83f2ab318c954848b2be15d0926b94fb in this term.
I20260619 08:54:59.700695 1924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "83f2ab318c954848b2be15d0926b94fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 863 } } ignore_live_leader: false dest_uuid: "aa6ea950b6234668969492322c29714b"
I20260619 08:54:59.700820 1923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "8380f1103150498b9eaebf58f4c49b7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 863 } } ignore_live_leader: false dest_uuid: "aa6ea950b6234668969492322c29714b"
I20260619 08:54:59.700847 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "83f2ab318c954848b2be15d0926b94fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 863 } } ignore_live_leader: false dest_uuid: "171a74bcea154d9497fdbc1dff677801"
I20260619 08:54:59.701032 1269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "83f2ab318c954848b2be15d0926b94fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 863 } } ignore_live_leader: false dest_uuid: "408d1f4ba9f245f1aa2608445e067a01"
I20260619 08:54:59.701109 1269 raft_consensus.cc:3037] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01 [term 1 FOLLOWER]: Advancing to term 2
I20260619 08:54:59.701092 1793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "83f2ab318c954848b2be15d0926b94fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 863 } } ignore_live_leader: false dest_uuid: "8380f1103150498b9eaebf58f4c49b7b"
I20260619 08:54:59.701079 1398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "83f2ab318c954848b2be15d0926b94fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 863 } } ignore_live_leader: false dest_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9"
I20260619 08:54:59.701155 1398 raft_consensus.cc:3037] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9 [term 1 FOLLOWER]: Advancing to term 2
I20260619 08:54:59.701169 1793 raft_consensus.cc:2379] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 83f2ab318c954848b2be15d0926b94fb in current term 2: Already voted for candidate 8380f1103150498b9eaebf58f4c49b7b in this term.
I20260619 08:54:59.701659 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "8380f1103150498b9eaebf58f4c49b7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 863 } } ignore_live_leader: false dest_uuid: "171a74bcea154d9497fdbc1dff677801"
I20260619 08:54:59.701920 1269 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f2ab318c954848b2be15d0926b94fb in term 2.
I20260619 08:54:59.702041 1398 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f2ab318c954848b2be15d0926b94fb in term 2.
I20260619 08:54:59.702180 1399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "8380f1103150498b9eaebf58f4c49b7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 863 } } ignore_live_leader: false dest_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9"
I20260619 08:54:59.702200 1269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "8380f1103150498b9eaebf58f4c49b7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 863 } } ignore_live_leader: false dest_uuid: "408d1f4ba9f245f1aa2608445e067a01"
I20260619 08:54:59.702275 1269 raft_consensus.cc:2379] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8380f1103150498b9eaebf58f4c49b7b in current term 2: Already voted for candidate 83f2ab318c954848b2be15d0926b94fb in this term.
I20260619 08:54:59.702277 1399 raft_consensus.cc:2379] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8380f1103150498b9eaebf58f4c49b7b in current term 2: Already voted for candidate 83f2ab318c954848b2be15d0926b94fb in this term.
I20260619 08:54:59.702447 1728 leader_election.cc:302] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 8380f1103150498b9eaebf58f4c49b7b; no voters: 171a74bcea154d9497fdbc1dff677801, 408d1f4ba9f245f1aa2608445e067a01, 83f2ab318c954848b2be15d0926b94fb, aa6ea950b6234668969492322c29714b
I20260619 08:54:59.702558 2112 raft_consensus.cc:2726] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 08:54:59.779801 392 test_util.cc:268] Using random seed: 1922720974
I20260619 08:54:59.944756 2129 raft_consensus.cc:482] T 63f80b9eea6c49659b4407c51e019b96 P 171a74bcea154d9497fdbc1dff677801 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b6329eab306a4d6fa208557bd278a5d7)
I20260619 08:54:59.944849 2129 raft_consensus.cc:504] T 63f80b9eea6c49659b4407c51e019b96 P 171a74bcea154d9497fdbc1dff677801 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:54:59.945348 2129 leader_election.cc:288] T 63f80b9eea6c49659b4407c51e019b96 P 171a74bcea154d9497fdbc1dff677801 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f2ab318c954848b2be15d0926b94fb (127.0.98.1:39977), 8380f1103150498b9eaebf58f4c49b7b (127.0.98.6:38855), aa6ea950b6234668969492322c29714b (127.0.98.7:41045), 408d1f4ba9f245f1aa2608445e067a01 (127.0.98.2:46549), b6329eab306a4d6fa208557bd278a5d7 (127.0.98.4:35389), 4e0e4dfff5e94d4b8e4b97882c68c5a9 (127.0.98.3:45507)
I20260619 08:54:59.950124 2136 raft_consensus.cc:482] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b6329eab306a4d6fa208557bd278a5d7)
I20260619 08:54:59.950219 1793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "171a74bcea154d9497fdbc1dff677801" candidate_term: 2 candidate_status { last_received { term: 1 index: 632 } } ignore_live_leader: false dest_uuid: "8380f1103150498b9eaebf58f4c49b7b" is_pre_election: true
I20260619 08:54:59.950309 1793 raft_consensus.cc:2379] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 171a74bcea154d9497fdbc1dff677801 in current term 2: Already voted for candidate 8380f1103150498b9eaebf58f4c49b7b in this term.
I20260619 08:54:59.950232 2136 raft_consensus.cc:504] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:54:59.950239 1399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "171a74bcea154d9497fdbc1dff677801" candidate_term: 2 candidate_status { last_received { term: 1 index: 632 } } ignore_live_leader: false dest_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" is_pre_election: true
I20260619 08:54:59.950354 1399 raft_consensus.cc:2379] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 171a74bcea154d9497fdbc1dff677801 in current term 2: Already voted for candidate 83f2ab318c954848b2be15d0926b94fb in this term.
I20260619 08:54:59.950884 1923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "171a74bcea154d9497fdbc1dff677801" candidate_term: 2 candidate_status { last_received { term: 1 index: 632 } } ignore_live_leader: false dest_uuid: "aa6ea950b6234668969492322c29714b" is_pre_election: true
I20260619 08:54:59.950979 1923 raft_consensus.cc:2396] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 171a74bcea154d9497fdbc1dff677801 for term 2 because replica has last-logged OpId of term: 1 index: 856, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 632.
I20260619 08:54:59.951597 2136 leader_election.cc:288] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83f2ab318c954848b2be15d0926b94fb (127.0.98.1:39977), 8380f1103150498b9eaebf58f4c49b7b (127.0.98.6:38855), 171a74bcea154d9497fdbc1dff677801 (127.0.98.5:38567), 408d1f4ba9f245f1aa2608445e067a01 (127.0.98.2:46549), b6329eab306a4d6fa208557bd278a5d7 (127.0.98.4:35389), 4e0e4dfff5e94d4b8e4b97882c68c5a9 (127.0.98.3:45507)
I20260619 08:54:59.951745 1137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "171a74bcea154d9497fdbc1dff677801" candidate_term: 2 candidate_status { last_received { term: 1 index: 632 } } ignore_live_leader: false dest_uuid: "83f2ab318c954848b2be15d0926b94fb" is_pre_election: true
I20260619 08:54:59.951833 1137 raft_consensus.cc:2379] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 171a74bcea154d9497fdbc1dff677801 in current term 2: Already voted for candidate 83f2ab318c954848b2be15d0926b94fb in this term.
I20260619 08:54:59.952235 1595 leader_election.cc:302] T 63f80b9eea6c49659b4407c51e019b96 P 171a74bcea154d9497fdbc1dff677801 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 171a74bcea154d9497fdbc1dff677801; no voters: 4e0e4dfff5e94d4b8e4b97882c68c5a9, 8380f1103150498b9eaebf58f4c49b7b, 83f2ab318c954848b2be15d0926b94fb, aa6ea950b6234668969492322c29714b
I20260619 08:54:59.952977 2129 raft_consensus.cc:3037] T 63f80b9eea6c49659b4407c51e019b96 P 171a74bcea154d9497fdbc1dff677801 [term 1 FOLLOWER]: Advancing to term 2
I20260619 08:54:59.953311 1269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "171a74bcea154d9497fdbc1dff677801" candidate_term: 2 candidate_status { last_received { term: 1 index: 632 } } ignore_live_leader: false dest_uuid: "408d1f4ba9f245f1aa2608445e067a01" is_pre_election: true
I20260619 08:54:59.953410 1269 raft_consensus.cc:2379] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 171a74bcea154d9497fdbc1dff677801 in current term 2: Already voted for candidate 83f2ab318c954848b2be15d0926b94fb in this term.
I20260619 08:54:59.954246 1137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "aa6ea950b6234668969492322c29714b" candidate_term: 2 candidate_status { last_received { term: 1 index: 856 } } ignore_live_leader: false dest_uuid: "83f2ab318c954848b2be15d0926b94fb" is_pre_election: true
I20260619 08:54:59.954329 1137 raft_consensus.cc:2379] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aa6ea950b6234668969492322c29714b in current term 2: Already voted for candidate 83f2ab318c954848b2be15d0926b94fb in this term.
I20260619 08:54:59.955425 1793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "aa6ea950b6234668969492322c29714b" candidate_term: 2 candidate_status { last_received { term: 1 index: 856 } } ignore_live_leader: false dest_uuid: "8380f1103150498b9eaebf58f4c49b7b" is_pre_election: true
I20260619 08:54:59.955524 1793 raft_consensus.cc:2379] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aa6ea950b6234668969492322c29714b in current term 2: Already voted for candidate 8380f1103150498b9eaebf58f4c49b7b in this term.
I20260619 08:54:59.955849 1269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "aa6ea950b6234668969492322c29714b" candidate_term: 2 candidate_status { last_received { term: 1 index: 856 } } ignore_live_leader: false dest_uuid: "408d1f4ba9f245f1aa2608445e067a01" is_pre_election: true
I20260619 08:54:59.955955 1269 raft_consensus.cc:2379] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aa6ea950b6234668969492322c29714b in current term 2: Already voted for candidate 83f2ab318c954848b2be15d0926b94fb in this term.
I20260619 08:54:59.957379 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "aa6ea950b6234668969492322c29714b" candidate_term: 2 candidate_status { last_received { term: 1 index: 856 } } ignore_live_leader: false dest_uuid: "171a74bcea154d9497fdbc1dff677801" is_pre_election: true
I20260619 08:54:59.958082 1399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "aa6ea950b6234668969492322c29714b" candidate_term: 2 candidate_status { last_received { term: 1 index: 856 } } ignore_live_leader: false dest_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" is_pre_election: true
I20260619 08:54:59.958169 1399 raft_consensus.cc:2379] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aa6ea950b6234668969492322c29714b in current term 2: Already voted for candidate 83f2ab318c954848b2be15d0926b94fb in this term.
I20260619 08:54:59.958357 1859 leader_election.cc:302] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: aa6ea950b6234668969492322c29714b; no voters: 408d1f4ba9f245f1aa2608445e067a01, 4e0e4dfff5e94d4b8e4b97882c68c5a9, 8380f1103150498b9eaebf58f4c49b7b, 83f2ab318c954848b2be15d0926b94fb
I20260619 08:54:59.958591 2136 raft_consensus.cc:3037] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [term 1 FOLLOWER]: Advancing to term 2
I20260619 08:54:59.966723 2129 raft_consensus.cc:2726] T 63f80b9eea6c49659b4407c51e019b96 P 171a74bcea154d9497fdbc1dff677801 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 08:54:59.966763 2136 raft_consensus.cc:2726] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 08:54:59.966845 1662 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 171a74bcea154d9497fdbc1dff677801 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa6ea950b6234668969492322c29714b in term 2.
I20260619 08:55:00.384323 392 test_util.cc:268] Using random seed: 1923325495
W20260619 08:55:01.068595 1204 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.0.98.4:35389 for RPC RequestConsensusVote timed out after 1.696s (ON_OUTBOUND_QUEUE)
I20260619 08:55:01.089267 392 test_util.cc:268] Using random seed: 1924030439
W20260619 08:55:01.230151 1073 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.0.98.4:35389 for RPC RequestConsensusVote timed out after 1.528s (ON_OUTBOUND_QUEUE)
W20260619 08:55:01.308580 1728 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.0.98.4:35389 for RPC RequestConsensusVote timed out after 1.703s (ON_OUTBOUND_QUEUE)
W20260619 08:55:01.336400 1335 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.0.98.4:35389 for RPC RequestConsensusVote timed out after 2.074s (ON_OUTBOUND_QUEUE)
I20260619 08:55:01.562027 2148 raft_consensus.cc:482] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 08:55:01.562269 2148 raft_consensus.cc:504] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:55:01.562564 2148 leader_election.cc:288] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83f2ab318c954848b2be15d0926b94fb (127.0.98.1:39977), 8380f1103150498b9eaebf58f4c49b7b (127.0.98.6:38855), 171a74bcea154d9497fdbc1dff677801 (127.0.98.5:38567), 408d1f4ba9f245f1aa2608445e067a01 (127.0.98.2:46549), b6329eab306a4d6fa208557bd278a5d7 (127.0.98.4:35389), 4e0e4dfff5e94d4b8e4b97882c68c5a9 (127.0.98.3:45507)
I20260619 08:55:01.562757 1137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "aa6ea950b6234668969492322c29714b" candidate_term: 3 candidate_status { last_received { term: 1 index: 856 } } ignore_live_leader: false dest_uuid: "83f2ab318c954848b2be15d0926b94fb" is_pre_election: true
I20260619 08:55:01.562755 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "aa6ea950b6234668969492322c29714b" candidate_term: 3 candidate_status { last_received { term: 1 index: 856 } } ignore_live_leader: false dest_uuid: "171a74bcea154d9497fdbc1dff677801" is_pre_election: true
I20260619 08:55:01.562765 1793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "aa6ea950b6234668969492322c29714b" candidate_term: 3 candidate_status { last_received { term: 1 index: 856 } } ignore_live_leader: false dest_uuid: "8380f1103150498b9eaebf58f4c49b7b" is_pre_election: true
I20260619 08:55:01.562870 1137 raft_consensus.cc:2396] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aa6ea950b6234668969492322c29714b for term 3 because replica has last-logged OpId of term: 1 index: 863, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 856.
I20260619 08:55:01.562906 1662 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 171a74bcea154d9497fdbc1dff677801 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa6ea950b6234668969492322c29714b in term 2.
I20260619 08:55:01.562923 1793 raft_consensus.cc:2396] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aa6ea950b6234668969492322c29714b for term 3 because replica has last-logged OpId of term: 1 index: 863, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 856.
I20260619 08:55:01.563249 1399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "aa6ea950b6234668969492322c29714b" candidate_term: 3 candidate_status { last_received { term: 1 index: 856 } } ignore_live_leader: false dest_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" is_pre_election: true
I20260619 08:55:01.563257 1269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "aa6ea950b6234668969492322c29714b" candidate_term: 3 candidate_status { last_received { term: 1 index: 856 } } ignore_live_leader: false dest_uuid: "408d1f4ba9f245f1aa2608445e067a01" is_pre_election: true
I20260619 08:55:01.563344 1399 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa6ea950b6234668969492322c29714b in term 2.
I20260619 08:55:01.563359 1269 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa6ea950b6234668969492322c29714b in term 2.
I20260619 08:55:01.563565 1859 leader_election.cc:302] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 171a74bcea154d9497fdbc1dff677801, 408d1f4ba9f245f1aa2608445e067a01, 4e0e4dfff5e94d4b8e4b97882c68c5a9, aa6ea950b6234668969492322c29714b; no voters: 8380f1103150498b9eaebf58f4c49b7b, 83f2ab318c954848b2be15d0926b94fb
I20260619 08:55:01.563700 2148 raft_consensus.cc:2781] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260619 08:55:01.563753 2148 raft_consensus.cc:482] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 08:55:01.563783 2148 raft_consensus.cc:3037] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [term 2 FOLLOWER]: Advancing to term 3
I20260619 08:55:01.564476 2148 raft_consensus.cc:504] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:55:01.564730 2148 leader_election.cc:288] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [CANDIDATE]: Term 3 election: Requested vote from peers 83f2ab318c954848b2be15d0926b94fb (127.0.98.1:39977), 8380f1103150498b9eaebf58f4c49b7b (127.0.98.6:38855), 171a74bcea154d9497fdbc1dff677801 (127.0.98.5:38567), 408d1f4ba9f245f1aa2608445e067a01 (127.0.98.2:46549), b6329eab306a4d6fa208557bd278a5d7 (127.0.98.4:35389), 4e0e4dfff5e94d4b8e4b97882c68c5a9 (127.0.98.3:45507)
I20260619 08:55:01.564744 1793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "aa6ea950b6234668969492322c29714b" candidate_term: 3 candidate_status { last_received { term: 1 index: 856 } } ignore_live_leader: false dest_uuid: "8380f1103150498b9eaebf58f4c49b7b"
I20260619 08:55:01.564806 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "aa6ea950b6234668969492322c29714b" candidate_term: 3 candidate_status { last_received { term: 1 index: 856 } } ignore_live_leader: false dest_uuid: "171a74bcea154d9497fdbc1dff677801"
I20260619 08:55:01.564849 1793 raft_consensus.cc:3037] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 2 FOLLOWER]: Advancing to term 3
I20260619 08:55:01.564873 1662 raft_consensus.cc:3037] T 63f80b9eea6c49659b4407c51e019b96 P 171a74bcea154d9497fdbc1dff677801 [term 2 FOLLOWER]: Advancing to term 3
I20260619 08:55:01.565110 1399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "aa6ea950b6234668969492322c29714b" candidate_term: 3 candidate_status { last_received { term: 1 index: 856 } } ignore_live_leader: false dest_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9"
I20260619 08:55:01.565171 1399 raft_consensus.cc:3037] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9 [term 2 FOLLOWER]: Advancing to term 3
I20260619 08:55:01.565709 1662 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 171a74bcea154d9497fdbc1dff677801 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa6ea950b6234668969492322c29714b in term 3.
I20260619 08:55:01.565804 1793 raft_consensus.cc:2396] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate aa6ea950b6234668969492322c29714b for term 3 because replica has last-logged OpId of term: 1 index: 863, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 856.
I20260619 08:55:01.565905 1269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "aa6ea950b6234668969492322c29714b" candidate_term: 3 candidate_status { last_received { term: 1 index: 856 } } ignore_live_leader: false dest_uuid: "408d1f4ba9f245f1aa2608445e067a01"
I20260619 08:55:01.565966 1399 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa6ea950b6234668969492322c29714b in term 3.
I20260619 08:55:01.565979 1269 raft_consensus.cc:3037] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01 [term 2 FOLLOWER]: Advancing to term 3
I20260619 08:55:01.566172 1137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "aa6ea950b6234668969492322c29714b" candidate_term: 3 candidate_status { last_received { term: 1 index: 856 } } ignore_live_leader: false dest_uuid: "83f2ab318c954848b2be15d0926b94fb"
I20260619 08:55:01.566242 1137 raft_consensus.cc:3037] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [term 2 FOLLOWER]: Advancing to term 3
I20260619 08:55:01.635031 1269 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa6ea950b6234668969492322c29714b in term 3.
I20260619 08:55:01.635087 1137 raft_consensus.cc:2396] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate aa6ea950b6234668969492322c29714b for term 3 because replica has last-logged OpId of term: 1 index: 863, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 856.
I20260619 08:55:01.635370 1859 leader_election.cc:302] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 171a74bcea154d9497fdbc1dff677801, 408d1f4ba9f245f1aa2608445e067a01, 4e0e4dfff5e94d4b8e4b97882c68c5a9, aa6ea950b6234668969492322c29714b; no voters: 8380f1103150498b9eaebf58f4c49b7b, 83f2ab318c954848b2be15d0926b94fb
I20260619 08:55:01.635564 2148 raft_consensus.cc:2781] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [term 3 FOLLOWER]: Leader election won for term 3
I20260619 08:55:01.635739 2148 raft_consensus.cc:686] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [term 3 LEADER]: Becoming Leader. State: Replica: aa6ea950b6234668969492322c29714b, State: Running, Role: LEADER
I20260619 08:55:01.635921 2148 consensus_queue.cc:237] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 855, Committed index: 855, Last appended: 1.856, Last appended by leader: 856, Current term: 3, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:55:01.636680 988 catalog_manager.cc:5495] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b reported cstate change: term changed from 1 to 3, leader changed from b6329eab306a4d6fa208557bd278a5d7 (127.0.98.4) to aa6ea950b6234668969492322c29714b (127.0.98.7). New cstate: current_term: 3 leader_uuid: "aa6ea950b6234668969492322c29714b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } health_report { overall_health: UNKNOWN } } }
W20260619 08:55:01.794304 1465 connection.cc:232] RPC call timeout handler was delayed by -2.9073458839999944s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1s for a response
W20260619 08:55:01.794523 1467 connection.cc:232] RPC call timeout handler was delayed by -2.7973591969999916s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1s for a response
W20260619 08:55:01.794561 1466 connection.cc:232] RPC call timeout handler was delayed by -3.1357593779999888s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1s for a response
W20260619 08:55:01.794603 1466 connection.cc:232] RPC call timeout handler was delayed by -3.0248316269999691s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1s for a response
I20260619 08:55:01.796363 2155 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:44860 (ReqId={client: cb41fa24abb24d9d90f9ea8ec345fa65, seq_no=855, attempt_no=0}) took 3699 ms. Trace:
I20260619 08:55:01.796583 2155 rpcz_store.cc:276] 0619 08:54:58.096363 (+ 0us) service_pool.cc:170] Inserting onto call queue
0619 08:54:58.096401 (+ 38us) service_pool.cc:227] Handling call
0619 08:55:01.796347 (+3699946us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0619 08:54:58.096456 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63f80b9eea6c49659b4407c51e019b96
0619 08:54:58.096463 (+ 7us) write_op.cc:432] Acquiring schema lock in shared mode
0619 08:54:58.096463 (+ 0us) write_op.cc:435] Acquired schema lock
0619 08:54:58.096463 (+ 0us) tablet.cc:660] Decoding operations
0619 08:54:58.096473 (+ 10us) write_op.cc:620] Acquiring the partition lock for write op
0619 08:54:58.096474 (+ 1us) write_op.cc:641] Partition lock acquired for write op
0619 08:54:58.096475 (+ 1us) tablet.cc:683] Acquiring locks for 10 operations
0619 08:54:58.096483 (+ 8us) tablet.cc:699] Row locks acquired
0619 08:54:58.096483 (+ 0us) write_op.cc:260] PREPARE: finished
0619 08:54:58.096486 (+ 3us) write_op.cc:270] Start()
0619 08:54:58.096487 (+ 1us) write_op.cc:276] Timestamp: P: 1781859298096479 usec, L: 0
0619 08:54:58.096488 (+ 1us) op_driver.cc:331] REPLICATION: starting
0619 08:54:58.096510 (+ 22us) log.cc:842] Serialized 696 byte log entry
0619 08:55:01.795314 (+3698804us) op_driver.cc:444] REPLICATION: finished
0619 08:55:01.795908 (+ 594us) write_op.cc:301] APPLY: starting
0619 08:55:01.795911 (+ 3us) tablet.cc:1365] starting BulkCheckPresence
0619 08:55:01.795917 (+ 6us) tablet.cc:1368] finished BulkCheckPresence
0619 08:55:01.795919 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0619 08:55:01.795942 (+ 23us) tablet.cc:1381] finished ApplyRowOperation cycle
0619 08:55:01.795946 (+ 4us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0619 08:55:01.795947 (+ 1us) write_op.cc:312] APPLY: finished
0619 08:55:01.795965 (+ 18us) log.cc:842] Serialized 77 byte log entry
0619 08:55:01.796320 (+ 355us) write_op.cc:489] Releasing partition, row and schema locks
0619 08:55:01.796326 (+ 6us) write_op.cc:454] Released schema lock
0619 08:55:01.796326 (+ 0us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":318,"mutex_wait_us":70,"num_ops":10,"prepare.queue_time_us":14,"prepare.run_cpu_time_us":121,"prepare.run_wall_time_us":248,"raft.queue_time_us":143,"raft.run_cpu_time_us":44,"raft.run_wall_time_us":44,"replication_time_us":3698823,"spinlock_wait_cycles":5376,"thread_start_us":402,"threads_started":2}]]}
I20260619 08:55:01.796855 1923 raft_consensus.cc:1226] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [term 3 LEADER]: Rejecting Update request from peer b6329eab306a4d6fa208557bd278a5d7 for earlier term 1. Current term is 3. Ops: [1.857-1.863]
I20260619 08:55:01.797341 2022 consensus_queue.cc:1046] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 }, Status: INVALID_TERM, Last received: 1.856, Next index: 857, Last known committed idx: 855, Time since last communication: 0.000s
I20260619 08:55:01.797448 2022 consensus_queue.cc:1225] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [LEADER]: Peer aa6ea950b6234668969492322c29714b log is divergent from this leader: its last log entry 3.857 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 855
I20260619 08:55:01.797487 2037 raft_consensus.cc:3032] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [term 1 LEADER]: Stepping down as leader of term 1
I20260619 08:55:01.797537 2037 raft_consensus.cc:729] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b6329eab306a4d6fa208557bd278a5d7, State: Running, Role: LEADER
I20260619 08:55:01.797588 2037 consensus_queue.cc:260] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 856, Committed index: 856, Last appended: 1.863, Last appended by leader: 863, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:55:01.798336 1137 raft_consensus.cc:1226] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [term 3 FOLLOWER]: Rejecting Update request from peer b6329eab306a4d6fa208557bd278a5d7 for earlier term 1. Current term is 3. Ops: []
I20260619 08:55:01.798537 1793 raft_consensus.cc:1226] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 3 FOLLOWER]: Rejecting Update request from peer b6329eab306a4d6fa208557bd278a5d7 for earlier term 1. Current term is 3. Ops: []
W20260619 08:55:01.798379 2162 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44860: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.004s ago.
I20260619 08:55:01.799124 2015 consensus_queue.cc:1046] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 }, Status: INVALID_TERM, Last received: 1.863, Next index: 864, Last known committed idx: 855, Time since last communication: 0.000s
I20260619 08:55:01.800107 1662 raft_consensus.cc:1226] T 63f80b9eea6c49659b4407c51e019b96 P 171a74bcea154d9497fdbc1dff677801 [term 3 FOLLOWER]: Rejecting Update request from peer b6329eab306a4d6fa208557bd278a5d7 for earlier term 1. Current term is 3. Ops: [1.633-1.863]
I20260619 08:55:01.804118 1269 raft_consensus.cc:1226] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01 [term 3 FOLLOWER]: Rejecting Update request from peer b6329eab306a4d6fa208557bd278a5d7 for earlier term 1. Current term is 3. Ops: [1.422-1.863]
I20260619 08:55:01.805308 2083 consensus_queue.cc:1046] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 }, Status: INVALID_TERM, Last received: 1.632, Next index: 633, Last known committed idx: 628, Time since last communication: 0.000s
I20260619 08:55:01.806502 2083 consensus_queue.cc:1046] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 }, Status: INVALID_TERM, Last received: 1.421, Next index: 422, Last known committed idx: 414, Time since last communication: 0.000s
I20260619 08:55:01.806700 2082 consensus_queue.cc:1046] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 }, Status: INVALID_TERM, Last received: 1.863, Next index: 864, Last known committed idx: 855, Time since last communication: 0.000s
I20260619 08:55:01.806921 1530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "408d1f4ba9f245f1aa2608445e067a01" candidate_term: 2 candidate_status { last_received { term: 1 index: 421 } } ignore_live_leader: false dest_uuid: "b6329eab306a4d6fa208557bd278a5d7" is_pre_election: true
I20260619 08:55:01.807507 1531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "83f2ab318c954848b2be15d0926b94fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 863 } } ignore_live_leader: false dest_uuid: "b6329eab306a4d6fa208557bd278a5d7"
I20260619 08:55:01.807616 1399 raft_consensus.cc:1226] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9 [term 3 FOLLOWER]: Rejecting Update request from peer b6329eab306a4d6fa208557bd278a5d7 for earlier term 1. Current term is 3. Ops: [1.237-1.863]
I20260619 08:55:01.807767 1073 leader_election.cc:302] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 408d1f4ba9f245f1aa2608445e067a01, 4e0e4dfff5e94d4b8e4b97882c68c5a9, 83f2ab318c954848b2be15d0926b94fb; no voters: 171a74bcea154d9497fdbc1dff677801, 8380f1103150498b9eaebf58f4c49b7b, aa6ea950b6234668969492322c29714b, b6329eab306a4d6fa208557bd278a5d7
I20260619 08:55:01.807817 1529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "83f2ab318c954848b2be15d0926b94fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 863 } } ignore_live_leader: false dest_uuid: "b6329eab306a4d6fa208557bd278a5d7" is_pre_election: true
I20260619 08:55:01.808408 2167 raft_consensus.cc:2726] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 08:55:01.809087 2037 raft_consensus.cc:3037] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [term 1 FOLLOWER]: Advancing to term 3
I20260619 08:55:01.809298 1531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "171a74bcea154d9497fdbc1dff677801" candidate_term: 2 candidate_status { last_received { term: 1 index: 632 } } ignore_live_leader: false dest_uuid: "b6329eab306a4d6fa208557bd278a5d7" is_pre_election: true
I20260619 08:55:01.809333 1529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 236 } } ignore_live_leader: false dest_uuid: "b6329eab306a4d6fa208557bd278a5d7" is_pre_election: true
I20260619 08:55:01.809405 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "8380f1103150498b9eaebf58f4c49b7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 863 } } ignore_live_leader: false dest_uuid: "b6329eab306a4d6fa208557bd278a5d7"
I20260619 08:55:01.809499 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "8380f1103150498b9eaebf58f4c49b7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 863 } } ignore_live_leader: false dest_uuid: "b6329eab306a4d6fa208557bd278a5d7" is_pre_election: true
I20260619 08:55:01.810611 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "aa6ea950b6234668969492322c29714b" candidate_term: 2 candidate_status { last_received { term: 1 index: 856 } } ignore_live_leader: false dest_uuid: "b6329eab306a4d6fa208557bd278a5d7" is_pre_election: true
I20260619 08:55:01.810637 1529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "aa6ea950b6234668969492322c29714b" candidate_term: 3 candidate_status { last_received { term: 1 index: 856 } } ignore_live_leader: false dest_uuid: "b6329eab306a4d6fa208557bd278a5d7"
I20260619 08:55:01.810644 1525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "aa6ea950b6234668969492322c29714b" candidate_term: 3 candidate_status { last_received { term: 1 index: 856 } } ignore_live_leader: false dest_uuid: "b6329eab306a4d6fa208557bd278a5d7" is_pre_election: true
W20260619 08:55:01.813001 1098 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51330: Illegal state: replica 83f2ab318c954848b2be15d0926b94fb is not leader of this config: current role FOLLOWER
W20260619 08:55:01.817960 1753 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49904: Illegal state: replica 8380f1103150498b9eaebf58f4c49b7b is not leader of this config: current role FOLLOWER
I20260619 08:55:01.827051 1399 raft_consensus.cc:1261] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9 [term 3 FOLLOWER]: Refusing update from remote peer aa6ea950b6234668969492322c29714b: Log matching property violated. Preceding OpId in replica: term: 1 index: 236. Preceding OpId from leader: term: 3 index: 858. (index mismatch)
I20260619 08:55:01.827087 1137 raft_consensus.cc:1261] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [term 3 FOLLOWER]: Refusing update from remote peer aa6ea950b6234668969492322c29714b: Log matching property violated. Preceding OpId in replica: term: 1 index: 863. Preceding OpId from leader: term: 3 index: 858. (term mismatch)
I20260619 08:55:01.827136 1137 pending_rounds.cc:76] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb: Aborting all ops after (but not including) 857
I20260619 08:55:01.827143 1793 raft_consensus.cc:1261] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 3 FOLLOWER]: Refusing update from remote peer aa6ea950b6234668969492322c29714b: Log matching property violated. Preceding OpId in replica: term: 1 index: 863. Preceding OpId from leader: term: 3 index: 858. (term mismatch)
I20260619 08:55:01.827185 1793 pending_rounds.cc:76] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b: Aborting all ops after (but not including) 857
I20260619 08:55:01.827158 1137 pending_rounds.cc:97] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb: Aborting uncommitted WRITE_OP operation due to leader change: 1.858
I20260619 08:55:01.827203 1793 pending_rounds.cc:97] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b: Aborting uncommitted WRITE_OP operation due to leader change: 1.858
I20260619 08:55:01.827250 1137 pending_rounds.cc:97] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb: Aborting uncommitted WRITE_OP operation due to leader change: 1.859
I20260619 08:55:01.827239 1269 raft_consensus.cc:1261] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01 [term 3 FOLLOWER]: Refusing update from remote peer aa6ea950b6234668969492322c29714b: Log matching property violated. Preceding OpId in replica: term: 1 index: 421. Preceding OpId from leader: term: 3 index: 858. (index mismatch)
I20260619 08:55:01.827281 1793 pending_rounds.cc:97] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b: Aborting uncommitted WRITE_OP operation due to leader change: 1.859
I20260619 08:55:01.827286 1137 pending_rounds.cc:97] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb: Aborting uncommitted WRITE_OP operation due to leader change: 1.860
I20260619 08:55:01.827322 1137 pending_rounds.cc:97] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb: Aborting uncommitted WRITE_OP operation due to leader change: 1.861
I20260619 08:55:01.827318 1793 pending_rounds.cc:97] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b: Aborting uncommitted WRITE_OP operation due to leader change: 1.860
I20260619 08:55:01.827353 1137 pending_rounds.cc:97] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb: Aborting uncommitted WRITE_OP operation due to leader change: 1.862
I20260619 08:55:01.827365 1793 pending_rounds.cc:97] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b: Aborting uncommitted WRITE_OP operation due to leader change: 1.861
I20260619 08:55:01.827385 1137 pending_rounds.cc:97] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb: Aborting uncommitted WRITE_OP operation due to leader change: 1.863
I20260619 08:55:01.827400 1793 pending_rounds.cc:97] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b: Aborting uncommitted WRITE_OP operation due to leader change: 1.862
I20260619 08:55:01.827430 1793 pending_rounds.cc:97] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b: Aborting uncommitted WRITE_OP operation due to leader change: 1.863
I20260619 08:55:01.827467 2148 consensus_queue.cc:1035] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 857, Last known committed idx: 230, Time since last communication: 0.000s
I20260619 08:55:01.827623 2171 consensus_queue.cc:1035] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [LEADER]: Connected to new peer: Peer: permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 857, Last known committed idx: 414, Time since last communication: 0.000s
I20260619 08:55:01.827839 1662 raft_consensus.cc:1261] T 63f80b9eea6c49659b4407c51e019b96 P 171a74bcea154d9497fdbc1dff677801 [term 3 FOLLOWER]: Refusing update from remote peer aa6ea950b6234668969492322c29714b: Log matching property violated. Preceding OpId in replica: term: 1 index: 632. Preceding OpId from leader: term: 3 index: 858. (index mismatch)
I20260619 08:55:01.828275 2149 consensus_queue.cc:1035] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 857, Last known committed idx: 855, Time since last communication: 0.000s
I20260619 08:55:01.828521 2149 consensus_queue.cc:1225] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [LEADER]: Peer 83f2ab318c954848b2be15d0926b94fb log is divergent from this leader: its last log entry 1.857 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 855
I20260619 08:55:01.828605 2172 consensus_queue.cc:1035] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 857, Last known committed idx: 855, Time since last communication: 0.000s
I20260619 08:55:01.828677 2172 consensus_queue.cc:1225] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [LEADER]: Peer 8380f1103150498b9eaebf58f4c49b7b log is divergent from this leader: its last log entry 1.857 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 855
I20260619 08:55:01.828823 2173 consensus_queue.cc:1035] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [LEADER]: Connected to new peer: Peer: permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 857, Last known committed idx: 628, Time since last communication: 0.000s
I20260619 08:55:01.828914 1793 raft_consensus.cc:1203] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 3 FOLLOWER]: Deduplicated request from leader. Original: 1.855->[1.856-3.858] Dedup: 1.856->[3.857-3.858]
I20260619 08:55:01.828958 1793 pending_rounds.cc:76] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b: Aborting all ops after (but not including) 856
I20260619 08:55:01.828975 1793 pending_rounds.cc:97] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b: Aborting uncommitted WRITE_OP operation due to leader change: 1.857
I20260619 08:55:01.829802 2175 raft_consensus.cc:482] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader aa6ea950b6234668969492322c29714b)
I20260619 08:55:01.829908 2175 raft_consensus.cc:504] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:55:01.830088 1137 raft_consensus.cc:1203] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb [term 3 FOLLOWER]: Deduplicated request from leader. Original: 1.855->[1.856-3.858] Dedup: 1.856->[3.857-3.858]
I20260619 08:55:01.830145 1137 pending_rounds.cc:76] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb: Aborting all ops after (but not including) 856
I20260619 08:55:01.830165 1137 pending_rounds.cc:97] T 63f80b9eea6c49659b4407c51e019b96 P 83f2ab318c954848b2be15d0926b94fb: Aborting uncommitted WRITE_OP operation due to leader change: 1.857
I20260619 08:55:01.830371 1138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "8380f1103150498b9eaebf58f4c49b7b" candidate_term: 4 candidate_status { last_received { term: 3 index: 858 } } ignore_live_leader: false dest_uuid: "83f2ab318c954848b2be15d0926b94fb" is_pre_election: true
I20260619 08:55:01.830406 1923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "8380f1103150498b9eaebf58f4c49b7b" candidate_term: 4 candidate_status { last_received { term: 3 index: 858 } } ignore_live_leader: false dest_uuid: "aa6ea950b6234668969492322c29714b" is_pre_election: true
I20260619 08:55:01.830812 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "8380f1103150498b9eaebf58f4c49b7b" candidate_term: 4 candidate_status { last_received { term: 3 index: 858 } } ignore_live_leader: false dest_uuid: "171a74bcea154d9497fdbc1dff677801" is_pre_election: true
I20260619 08:55:01.830869 2175 leader_election.cc:288] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83f2ab318c954848b2be15d0926b94fb (127.0.98.1:39977), aa6ea950b6234668969492322c29714b (127.0.98.7:41045), 171a74bcea154d9497fdbc1dff677801 (127.0.98.5:38567), 408d1f4ba9f245f1aa2608445e067a01 (127.0.98.2:46549), b6329eab306a4d6fa208557bd278a5d7 (127.0.98.4:35389), 4e0e4dfff5e94d4b8e4b97882c68c5a9 (127.0.98.3:45507)
I20260619 08:55:01.830886 1662 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 171a74bcea154d9497fdbc1dff677801 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8380f1103150498b9eaebf58f4c49b7b in term 3.
I20260619 08:55:01.830921 1268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "8380f1103150498b9eaebf58f4c49b7b" candidate_term: 4 candidate_status { last_received { term: 3 index: 858 } } ignore_live_leader: false dest_uuid: "408d1f4ba9f245f1aa2608445e067a01" is_pre_election: true
I20260619 08:55:01.830991 1268 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8380f1103150498b9eaebf58f4c49b7b in term 3.
I20260619 08:55:01.831120 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "8380f1103150498b9eaebf58f4c49b7b" candidate_term: 4 candidate_status { last_received { term: 3 index: 858 } } ignore_live_leader: false dest_uuid: "b6329eab306a4d6fa208557bd278a5d7" is_pre_election: true
I20260619 08:55:01.834582 1399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "8380f1103150498b9eaebf58f4c49b7b" candidate_term: 4 candidate_status { last_received { term: 3 index: 858 } } ignore_live_leader: false dest_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" is_pre_election: true
I20260619 08:55:01.834677 1399 raft_consensus.cc:2454] T 63f80b9eea6c49659b4407c51e019b96 P 4e0e4dfff5e94d4b8e4b97882c68c5a9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8380f1103150498b9eaebf58f4c49b7b in term 3.
I20260619 08:55:01.834884 1728 leader_election.cc:302] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 7 responses out of 7 voters: 4 yes votes; 3 no votes. yes voters: 171a74bcea154d9497fdbc1dff677801, 408d1f4ba9f245f1aa2608445e067a01, 4e0e4dfff5e94d4b8e4b97882c68c5a9, 8380f1103150498b9eaebf58f4c49b7b; no voters: 83f2ab318c954848b2be15d0926b94fb, aa6ea950b6234668969492322c29714b, b6329eab306a4d6fa208557bd278a5d7
I20260619 08:55:01.835044 2175 raft_consensus.cc:2781] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260619 08:55:01.835081 2175 raft_consensus.cc:482] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 3 FOLLOWER]: Starting leader election (detected failure of leader aa6ea950b6234668969492322c29714b)
I20260619 08:55:01.835103 2175 raft_consensus.cc:3037] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 3 FOLLOWER]: Advancing to term 4
I20260619 08:55:01.864787 2175 raft_consensus.cc:504] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:55:01.865057 2175 leader_election.cc:288] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [CANDIDATE]: Term 4 election: Requested vote from peers 83f2ab318c954848b2be15d0926b94fb (127.0.98.1:39977), aa6ea950b6234668969492322c29714b (127.0.98.7:41045), 171a74bcea154d9497fdbc1dff677801 (127.0.98.5:38567), 408d1f4ba9f245f1aa2608445e067a01 (127.0.98.2:46549), b6329eab306a4d6fa208557bd278a5d7 (127.0.98.4:35389), 4e0e4dfff5e94d4b8e4b97882c68c5a9 (127.0.98.3:45507)
I20260619 08:55:01.865182 1923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "8380f1103150498b9eaebf58f4c49b7b" candidate_term: 4 candidate_status { last_received { term: 3 index: 858 } } ignore_live_leader: false dest_uuid: "aa6ea950b6234668969492322c29714b"
I20260619 08:55:01.865614 1137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "8380f1103150498b9eaebf58f4c49b7b" candidate_term: 4 candidate_status { last_received { term: 3 index: 858 } } ignore_live_leader: false dest_uuid: "83f2ab318c954848b2be15d0926b94fb"
I20260619 08:55:01.865947 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "8380f1103150498b9eaebf58f4c49b7b" candidate_term: 4 candidate_status { last_received { term: 3 index: 858 } } ignore_live_leader: false dest_uuid: "171a74bcea154d9497fdbc1dff677801"
I20260619 08:55:01.866063 1524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "8380f1103150498b9eaebf58f4c49b7b" candidate_term: 4 candidate_status { last_received { term: 3 index: 858 } } ignore_live_leader: false dest_uuid: "b6329eab306a4d6fa208557bd278a5d7"
I20260619 08:55:01.866357 1269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "8380f1103150498b9eaebf58f4c49b7b" candidate_term: 4 candidate_status { last_received { term: 3 index: 858 } } ignore_live_leader: false dest_uuid: "408d1f4ba9f245f1aa2608445e067a01"
I20260619 08:55:01.866611 1793 raft_consensus.cc:1226] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 4 FOLLOWER]: Rejecting Update request from peer aa6ea950b6234668969492322c29714b for earlier term 3. Current term is 4. Ops: []
I20260619 08:55:01.867123 1728 leader_election.cc:302] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 8380f1103150498b9eaebf58f4c49b7b; no voters: 171a74bcea154d9497fdbc1dff677801, 408d1f4ba9f245f1aa2608445e067a01, 83f2ab318c954848b2be15d0926b94fb, aa6ea950b6234668969492322c29714b
I20260619 08:55:01.867255 2172 consensus_queue.cc:1046] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 }, Status: INVALID_TERM, Last received: 3.858, Next index: 859, Last known committed idx: 855, Time since last communication: 0.000s
I20260619 08:55:01.867353 2172 raft_consensus.cc:3032] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [term 3 LEADER]: Stepping down as leader of term 3
I20260619 08:55:01.867379 2172 raft_consensus.cc:729] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: aa6ea950b6234668969492322c29714b, State: Running, Role: LEADER
I20260619 08:55:01.867429 2172 consensus_queue.cc:260] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 858, Committed index: 858, Last appended: 3.859, Last appended by leader: 859, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83f2ab318c954848b2be15d0926b94fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39977 } } peers { permanent_uuid: "8380f1103150498b9eaebf58f4c49b7b" member_type: VOTER last_known_addr { host: "127.0.98.6" port: 38855 } } peers { permanent_uuid: "aa6ea950b6234668969492322c29714b" member_type: VOTER last_known_addr { host: "127.0.98.7" port: 41045 } } peers { permanent_uuid: "171a74bcea154d9497fdbc1dff677801" member_type: VOTER last_known_addr { host: "127.0.98.5" port: 38567 } } peers { permanent_uuid: "408d1f4ba9f245f1aa2608445e067a01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "b6329eab306a4d6fa208557bd278a5d7" member_type: VOTER last_known_addr { host: "127.0.98.4" port: 35389 } } peers { permanent_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45507 } }
I20260619 08:55:01.867542 2172 raft_consensus.cc:3037] T 63f80b9eea6c49659b4407c51e019b96 P aa6ea950b6234668969492322c29714b [term 3 FOLLOWER]: Advancing to term 4
I20260619 08:55:01.868896 2175 raft_consensus.cc:2726] T 63f80b9eea6c49659b4407c51e019b96 P 8380f1103150498b9eaebf58f4c49b7b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260619 08:55:01.869033 1399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63f80b9eea6c49659b4407c51e019b96" candidate_uuid: "8380f1103150498b9eaebf58f4c49b7b" candidate_term: 4 candidate_status { last_received { term: 3 index: 858 } } ignore_live_leader: false dest_uuid: "4e0e4dfff5e94d4b8e4b97882c68c5a9"
I20260619 08:55:01.960250 1530 raft_consensus.cc:2354] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 408d1f4ba9f245f1aa2608445e067a01 for earlier term 2. Current term is 3.
I20260619 08:55:01.960642 1529 raft_consensus.cc:1261] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [term 3 FOLLOWER]: Refusing update from remote peer aa6ea950b6234668969492322c29714b: Log matching property violated. Preceding OpId in replica: term: 1 index: 863. Preceding OpId from leader: term: 3 index: 858. (term mismatch)
I20260619 08:55:01.960708 1529 pending_rounds.cc:76] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7: Aborting all ops after (but not including) 857
I20260619 08:55:01.960731 1529 pending_rounds.cc:97] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7: Aborting uncommitted WRITE_OP operation due to leader change: 1.858
I20260619 08:55:01.960779 1204 leader_election.cc:398] T 63f80b9eea6c49659b4407c51e019b96 P 408d1f4ba9f245f1aa2608445e067a01 [CANDIDATE]: Term 2 pre-election: Vote denied by peer b6329eab306a4d6fa208557bd278a5d7 (127.0.98.4:35389) with higher term. Message: Invalid argument: T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 408d1f4ba9f245f1aa2608445e067a01 for earlier term 2. Current term is 3.
W20260619 08:55:01.960659 1525 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:44872 (request call id 13) took 146 ms (client timeout 100 ms). Trace:
W20260619 08:55:01.960824 1529 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44860: Aborted: Op aborted by new leader
I20260619 08:55:01.960875 1529 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:44860 (ReqId={client: cb41fa24abb24d9d90f9ea8ec345fa65, seq_no=856, attempt_no=0}) took 3864 ms. Trace:
W20260619 08:55:01.960852 1525 rpcz_store.cc:269] 0619 08:55:01.814140 (+ 0us) service_pool.cc:170] Inserting onto call queue
0619 08:55:01.814161 (+ 21us) service_pool.cc:227] Handling call
0619 08:55:01.960607 (+146446us) spinlock_profiling.cc:243] Waited 146 ms on lock 0xcf26fa4. stack: 00007f25e26aef1e 00007f25e786edac 00007f25e7d56149 00007f25e638ea34 00007f25e638fa88 00007f25e26c0953 00007f25e35246da 00007f25df2fb61e
0619 08:55:01.960641 (+ 34us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":336301312}
I20260619 08:55:01.960909 1529 rpcz_store.cc:276] 0619 08:54:58.096383 (+ 0us) service_pool.cc:170] Inserting onto call queue
0619 08:54:58.096464 (+ 81us) service_pool.cc:227] Handling call
0619 08:55:01.960872 (+3864408us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0619 08:54:58.096773 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63f80b9eea6c49659b4407c51e019b96
0619 08:54:58.096777 (+ 4us) write_op.cc:432] Acquiring schema lock in shared mode
0619 08:54:58.096777 (+ 0us) write_op.cc:435] Acquired schema lock
0619 08:54:58.096778 (+ 1us) tablet.cc:660] Decoding operations
0619 08:54:58.096787 (+ 9us) write_op.cc:620] Acquiring the partition lock for write op
0619 08:54:58.096788 (+ 1us) write_op.cc:641] Partition lock acquired for write op
0619 08:54:58.096788 (+ 0us) tablet.cc:683] Acquiring locks for 10 operations
0619 08:54:58.096797 (+ 9us) tablet.cc:699] Row locks acquired
0619 08:54:58.096798 (+ 1us) write_op.cc:260] PREPARE: finished
0619 08:54:58.096800 (+ 2us) write_op.cc:270] Start()
0619 08:54:58.096801 (+ 1us) write_op.cc:276] Timestamp: P: 1781859298096793 usec, L: 0
0619 08:54:58.096802 (+ 1us) op_driver.cc:331] REPLICATION: starting
0619 08:54:58.096814 (+ 12us) log.cc:842] Serialized 696 byte log entry
0619 08:55:01.960805 (+3863991us) op_driver.cc:444] REPLICATION: finished
0619 08:55:01.960807 (+ 2us) op_driver.cc:372] HandleFailure(Aborted: Op aborted by new leader)
0619 08:55:01.960809 (+ 2us) write_op.cc:489] Releasing partition, row and schema locks
0619 08:55:01.960821 (+ 12us) write_op.cc:454] Released schema lock
0619 08:55:01.960822 (+ 1us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":10,"prepare.queue_time_us":273,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":46,"replication_time_us":3863993}]]}
I20260619 08:55:01.961026 1529 pending_rounds.cc:97] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7: Aborting uncommitted WRITE_OP operation due to leader change: 1.859
W20260619 08:55:01.961064 1529 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44860: Aborted: Op aborted by new leader
I20260619 08:55:01.961107 1529 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:44860 (ReqId={client: cb41fa24abb24d9d90f9ea8ec345fa65, seq_no=854, attempt_no=0}) took 3864 ms. Trace:
I20260619 08:55:01.961133 1529 rpcz_store.cc:276] 0619 08:54:58.096338 (+ 0us) service_pool.cc:170] Inserting onto call queue
0619 08:54:58.096368 (+ 30us) service_pool.cc:227] Handling call
0619 08:55:01.961105 (+3864737us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0619 08:54:58.096822 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63f80b9eea6c49659b4407c51e019b96
0619 08:54:58.096825 (+ 3us) write_op.cc:432] Acquiring schema lock in shared mode
0619 08:54:58.096826 (+ 1us) write_op.cc:435] Acquired schema lock
0619 08:54:58.096826 (+ 0us) tablet.cc:660] Decoding operations
0619 08:54:58.096835 (+ 9us) write_op.cc:620] Acquiring the partition lock for write op
0619 08:54:58.096836 (+ 1us) write_op.cc:641] Partition lock acquired for write op
0619 08:54:58.096836 (+ 0us) tablet.cc:683] Acquiring locks for 10 operations
0619 08:54:58.096844 (+ 8us) tablet.cc:699] Row locks acquired
0619 08:54:58.096844 (+ 0us) write_op.cc:260] PREPARE: finished
0619 08:54:58.096846 (+ 2us) write_op.cc:270] Start()
0619 08:54:58.096848 (+ 2us) write_op.cc:276] Timestamp: P: 1781859298096840 usec, L: 0
0619 08:54:58.096848 (+ 0us) op_driver.cc:331] REPLICATION: starting
0619 08:54:58.096861 (+ 13us) log.cc:842] Serialized 696 byte log entry
0619 08:55:01.961055 (+3864194us) op_driver.cc:444] REPLICATION: finished
0619 08:55:01.961057 (+ 2us) op_driver.cc:372] HandleFailure(Aborted: Op aborted by new leader)
0619 08:55:01.961057 (+ 0us) write_op.cc:489] Releasing partition, row and schema locks
0619 08:55:01.961062 (+ 5us) write_op.cc:454] Released schema lock
0619 08:55:01.961063 (+ 1us) write_op.cc:335] FINISH: Op aborted
Metrics: {"spinlock_wait_cycles":768,"child_traces":[["op",{"mutex_wait_us":64,"num_ops":10,"prepare.queue_time_us":380,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"replication_time_us":3864203}]]}
I20260619 08:55:01.961242 1529 pending_rounds.cc:97] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7: Aborting uncommitted WRITE_OP operation due to leader change: 1.860
W20260619 08:55:01.961293 1529 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44860: Aborted: Op aborted by new leader
I20260619 08:55:01.961334 1529 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:44860 (ReqId={client: cb41fa24abb24d9d90f9ea8ec345fa65, seq_no=859, attempt_no=0}) took 3863 ms. Trace:
I20260619 08:55:01.961390 1529 rpcz_store.cc:276] 0619 08:54:58.097801 (+ 0us) service_pool.cc:170] Inserting onto call queue
0619 08:54:58.097831 (+ 30us) service_pool.cc:227] Handling call
0619 08:55:01.961333 (+3863502us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0619 08:54:58.097896 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63f80b9eea6c49659b4407c51e019b96
0619 08:54:58.097919 (+ 23us) write_op.cc:432] Acquiring schema lock in shared mode
0619 08:54:58.097919 (+ 0us) write_op.cc:435] Acquired schema lock
0619 08:54:58.097920 (+ 1us) tablet.cc:660] Decoding operations
0619 08:54:58.097929 (+ 9us) write_op.cc:620] Acquiring the partition lock for write op
0619 08:54:58.097929 (+ 0us) write_op.cc:641] Partition lock acquired for write op
0619 08:54:58.097930 (+ 1us) tablet.cc:683] Acquiring locks for 10 operations
0619 08:54:58.097937 (+ 7us) tablet.cc:699] Row locks acquired
0619 08:54:58.097937 (+ 0us) write_op.cc:260] PREPARE: finished
0619 08:54:58.097939 (+ 2us) write_op.cc:270] Start()
0619 08:54:58.097941 (+ 2us) write_op.cc:276] Timestamp: P: 1781859298097932 usec, L: 0
0619 08:54:58.097941 (+ 0us) op_driver.cc:331] REPLICATION: starting
0619 08:54:58.097974 (+ 33us) log.cc:842] Serialized 686 byte log entry
0619 08:55:01.961272 (+3863298us) op_driver.cc:444] REPLICATION: finished
0619 08:55:01.961273 (+ 1us) op_driver.cc:372] HandleFailure(Aborted: Op aborted by new leader)
0619 08:55:01.961273 (+ 0us) write_op.cc:489] Releasing partition, row and schema locks
0619 08:55:01.961292 (+ 19us) write_op.cc:454] Released schema lock
0619 08:55:01.961292 (+ 0us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":10,"prepare.queue_time_us":19,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":91,"replication_time_us":3863328}]]}
I20260619 08:55:01.961532 1529 pending_rounds.cc:97] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7: Aborting uncommitted WRITE_OP operation due to leader change: 1.861
W20260619 08:55:01.961575 1529 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44860: Aborted: Op aborted by new leader
I20260619 08:55:01.961637 1529 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:44860 (ReqId={client: cb41fa24abb24d9d90f9ea8ec345fa65, seq_no=861, attempt_no=0}) took 3863 ms. Trace:
I20260619 08:55:01.961673 1529 rpcz_store.cc:276] 0619 08:54:58.097816 (+ 0us) service_pool.cc:170] Inserting onto call queue
0619 08:54:58.097838 (+ 22us) service_pool.cc:227] Handling call
0619 08:55:01.961635 (+3863797us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0619 08:54:58.097990 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63f80b9eea6c49659b4407c51e019b96
0619 08:54:58.097993 (+ 3us) write_op.cc:432] Acquiring schema lock in shared mode
0619 08:54:58.097994 (+ 1us) write_op.cc:435] Acquired schema lock
0619 08:54:58.097994 (+ 0us) tablet.cc:660] Decoding operations
0619 08:54:58.098001 (+ 7us) write_op.cc:620] Acquiring the partition lock for write op
0619 08:54:58.098002 (+ 1us) write_op.cc:641] Partition lock acquired for write op
0619 08:54:58.098002 (+ 0us) tablet.cc:683] Acquiring locks for 10 operations
0619 08:54:58.098007 (+ 5us) tablet.cc:699] Row locks acquired
0619 08:54:58.098007 (+ 0us) write_op.cc:260] PREPARE: finished
0619 08:54:58.098009 (+ 2us) write_op.cc:270] Start()
0619 08:54:58.098011 (+ 2us) write_op.cc:276] Timestamp: P: 1781859298098002 usec, L: 0
0619 08:54:58.098011 (+ 0us) op_driver.cc:331] REPLICATION: starting
0619 08:54:58.098023 (+ 12us) log.cc:842] Serialized 686 byte log entry
0619 08:55:01.961565 (+3863542us) op_driver.cc:444] REPLICATION: finished
0619 08:55:01.961566 (+ 1us) op_driver.cc:372] HandleFailure(Aborted: Op aborted by new leader)
0619 08:55:01.961567 (+ 1us) write_op.cc:489] Releasing partition, row and schema locks
0619 08:55:01.961572 (+ 5us) write_op.cc:454] Released schema lock
0619 08:55:01.961573 (+ 1us) write_op.cc:335] FINISH: Op aborted
Metrics: {"spinlock_wait_cycles":16512,"child_traces":[["op",{"mutex_wait_us":48,"num_ops":10,"prepare.queue_time_us":115,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":37,"replication_time_us":3863550}]]}
I20260619 08:55:01.961783 1529 pending_rounds.cc:97] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7: Aborting uncommitted WRITE_OP operation due to leader change: 1.862
W20260619 08:55:01.961822 1529 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44860: Aborted: Op aborted by new leader
I20260619 08:55:01.961863 1529 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:44860 (ReqId={client: cb41fa24abb24d9d90f9ea8ec345fa65, seq_no=860, attempt_no=0}) took 3864 ms. Trace:
I20260619 08:55:01.961880 1529 rpcz_store.cc:276] 0619 08:54:58.097835 (+ 0us) service_pool.cc:170] Inserting onto call queue
0619 08:54:58.097923 (+ 88us) service_pool.cc:227] Handling call
0619 08:55:01.961862 (+3863939us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0619 08:54:58.098031 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63f80b9eea6c49659b4407c51e019b96
0619 08:54:58.098033 (+ 2us) write_op.cc:432] Acquiring schema lock in shared mode
0619 08:54:58.098034 (+ 1us) write_op.cc:435] Acquired schema lock
0619 08:54:58.098034 (+ 0us) tablet.cc:660] Decoding operations
0619 08:54:58.098041 (+ 7us) write_op.cc:620] Acquiring the partition lock for write op
0619 08:54:58.098041 (+ 0us) write_op.cc:641] Partition lock acquired for write op
0619 08:54:58.098041 (+ 0us) tablet.cc:683] Acquiring locks for 10 operations
0619 08:54:58.098045 (+ 4us) tablet.cc:699] Row locks acquired
0619 08:54:58.098045 (+ 0us) write_op.cc:260] PREPARE: finished
0619 08:54:58.098047 (+ 2us) write_op.cc:270] Start()
0619 08:54:58.098048 (+ 1us) write_op.cc:276] Timestamp: P: 1781859298098040 usec, L: 0
0619 08:54:58.098049 (+ 1us) op_driver.cc:331] REPLICATION: starting
0619 08:54:58.098058 (+ 9us) log.cc:842] Serialized 696 byte log entry
0619 08:55:01.961813 (+3863755us) op_driver.cc:444] REPLICATION: finished
0619 08:55:01.961815 (+ 2us) op_driver.cc:372] HandleFailure(Aborted: Op aborted by new leader)
0619 08:55:01.961815 (+ 0us) write_op.cc:489] Releasing partition, row and schema locks
0619 08:55:01.961820 (+ 5us) write_op.cc:454] Released schema lock
0619 08:55:01.961821 (+ 1us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":10,"prepare.queue_time_us":85,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"raft.queue_time_us":18,"raft.run_cpu_time_us":2,"raft.run_wall_time_us":2,"replication_time_us":3863761}]]}
I20260619 08:55:01.961980 1529 pending_rounds.cc:97] T 63f80b9eea6c49659b4407c51e019b96 P b6329eab306a4d6fa208557bd278a5d7: Aborting uncommitted WRITE_OP operation due to leader change: 1.863
W20260619 08:55:01.962015 1529 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44860: Aborted: Op aborted by new leader
I20260619 08:55:01.962046 1529 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:44860 (ReqId={client: cb41fa24abb24d9d90f9ea8ec345fa65, seq_no=858, attempt_no=0}) took 3864 ms. Trace:
I20260619 08:55:01.962073 1529 rpcz_store.cc:276] 0619 08:54:58.097784 (+ 0us) service_pool.cc:170] Inserting onto call queue
0619 08:54:58.097968 (+ 184us) service_pool.cc:227] Handling call
0619 08:55:01.962045 (+3864077us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0619 08:54:58.098078 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 63f80b9eea6c49659b4407c51e019b96
0619 08:54:58.098080 (+ 2us) write_op.cc:432] Acquiring schema lock in shared mode
0619 08:54:58.098080 (+ 0us) write_op.cc:435] Acquired schema lock
0619 08:54:58.098080 (+ 0us) tablet.cc:660] Decoding operations
0619 08:54:58.098086 (+ 6us) write_op.cc:620] Acquiring the partition lock for write op
0619 08:54:58.098086 (+ 0us) write_op.cc:641] Partition lock acquired for write op
0619 08:54:58.098087 (+ 1us) tablet.cc:683] Acquiring locks for 10 operations
0619 08:54:58.098091 (+ 4us) tablet.cc:699] Row locks acquired
0619 08:54:58.098091 (+ 0us) write_op.cc:260] PREPARE: finished
0619 08:54:58.098092 (+ 1us) write_op.cc:270] Start()
0619 08:54:58.098093 (+ 1us) write_op.cc:276] Timestamp: P: 1781859298098086 usec, L: 0
0619 08:54:58.098093 (+ 0us) op_driver.cc:331] REPLICATION: starting
0619 08:54:58.098104 (+ 11us) log.cc:842] Serialized 696 byte log entry
0619 08:55:01.962007 (+3863903us) op_driver.cc:444] REPLICATION: finished
0619 08:55:01.962008 (+ 1us) op_driver.cc:372] HandleFailure(Aborted: Op aborted by new leader)
0619 08:55:01.962009 (+ 1us) write_op.cc:489] Releasing partition, row and schema locks
0619 08:55:01.962013 (+ 4us) write_op.cc:454] Released schema lock
0619 08:55:01.962014 (+ 1us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":10,"prepare.queue_time_us":88,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"replication_time_us":3863911}]]}
W20260619 08:55:01.963735 1884 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42588: Illegal state: replica aa6ea950b6234668969492322c29714b is not leader of this config: current role FOLLOWER
W20260619 08:55:01.964072 1883 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42588: Illegal state: replica aa6ea950b6234668969492322c29714b is not leader of this config: current role FOLLOWER
W20260619 08:55:01.965826 1883 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42588: Illegal state: replica aa6ea950b6234668969492322c29714b is not leader of this config: current role FOLLOWER
W20260619 08:55:01.965858 1884 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42588: Illegal state: replica aa6ea950b6234668969492322c29714b is not leader of this config: current role FOLLOWER
W20260619 08:55:01.966976 1883 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42588: Illegal state: replica aa6ea950b6234668969492322c29714b is not leader of this config: current role FOLLOWER
W20260619 08:55:01.967001 1884 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42588: Illegal state: replica aa6ea950b6234668969492322c29714b is not leader of this config: current role FOLLOWER
W20260619 08:55:01.970711 1095 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51330: Illegal state: replica 83f2ab318c954848b2be15d0926b94fb is not leader of this config: current role FOLLOWER
W20260619 08:55:01.970700 1098 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51330: Illegal state: replica 83f2ab318c954848b2be15d0926b94fb is not leader of this config: current role FOLLOWER
W20260619 08:55:01.970758 1096 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51330: Illegal state: replica 83f2ab318c954848b2be15d0926b94fb is not leader of this config: current role FOLLOWER
W20260619 08:55:01.970711 1094 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51330: Illegal state: replica 83f2ab318c954848b2be15d0926b94fb is not leader of this config: current role FOLLOWER
W20260619 08:55:01.970681 1097 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51330: Illegal state: replica 83f2ab318c954848b2be15d0926b94fb is not leader of this config: current role FOLLOWER
W20260619 08:55:01.972865 1097 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:51330: Illegal state: replica 83f2ab318c954848b2be15d0926b94fb is not leader of this config: current role FOLLOWER
W20260619 08:55:01.974400 1753 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49904: Illegal state: replica 8380f1103150498b9eaebf58f4c49b7b is not leader of this config: current role FOLLOWER
W20260619 08:55:01.975493 1753 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49904: Illegal state: replica 8380f1103150498b9eaebf58f4c49b7b is not leader of this config: current role FOLLOWER
W20260619 08:55:01.975507 1752 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49904: Illegal state: replica 8380f1103150498b9eaebf58f4c49b7b is not leader of this config: current role FOLLOWER
W20260619 08:55:01.976614 1753 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49904: Illegal state: replica 8380f1103150498b9eaebf58f4c49b7b is not leader of this config: current role FOLLOWER
W20260619 08:55:01.976614 1752 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49904: Illegal state: replica 8380f1103150498b9eaebf58f4c49b7b is not leader of this config: current role FOLLOWER
W20260619 08:55:01.978732 1753 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49904: Illegal state: replica 8380f1103150498b9eaebf58f4c49b7b is not leader of this config: current role FOLLOWER
W20260619 08:55:01.984017 1620 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57468: Illegal state: replica 171a74bcea154d9497fdbc1dff677801 is not leader of this config: current role FOLLOWER
W20260619 08:55:01.984087 1621 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57468: Illegal state: replica 171a74bcea154d9497fdbc1dff677801 is not leader of this config: current role FOLLOWER
W20260619 08:55:01.984020 1622 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57468: Illegal state: replica 171a74bcea154d9497fdbc1dff677801 is not leader of this config: current role FOLLOWER
W20260619 08:55:01.984246 1619 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57468: Illegal state: replica 171a74bcea154d9497fdbc1dff677801 is not leader of this config: current role FOLLOWER
W20260619 08:55:01.984385 1618 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57468: Illegal state: replica 171a74bcea154d9497fdbc1dff677801 is not leader of this config: current role FOLLOWER
W20260619 08:55:01.986941 1618 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57468: Illegal state: replica 171a74bcea154d9497fdbc1dff677801 is not leader of this config: current role FOLLOWER
F20260619 08:55:01.989598 392 raft_consensus-itest.cc:398] Check failed: _s.ok() Bad status: Not found: leader replica not found
*** Check failure stack trace: ***
*** Aborted at 1781859301 (unix time) try "date -d @1781859301" if you are using GNU date ***
W20260619 08:55:01.993638 1229 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44186: Illegal state: replica 408d1f4ba9f245f1aa2608445e067a01 is not leader of this config: current role FOLLOWER
W20260619 08:55:01.993638 1227 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44186: Illegal state: replica 408d1f4ba9f245f1aa2608445e067a01 is not leader of this config: current role FOLLOWER
PC: @ 0x0 (unknown)
W20260619 08:55:01.993737 1226 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44186: Illegal state: replica 408d1f4ba9f245f1aa2608445e067a01 is not leader of this config: current role FOLLOWER
W20260619 08:55:01.993636 1228 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44186: Illegal state: replica 408d1f4ba9f245f1aa2608445e067a01 is not leader of this config: current role FOLLOWER
*** SIGABRT (@0x3e800000188) received by PID 392 (TID 0x7f32d9895440) from PID 392; stack trace: ***
W20260619 08:55:01.994534 1228 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44186: Illegal state: replica 408d1f4ba9f245f1aa2608445e067a01 is not leader of this config: current role FOLLOWER
@ 0x7f32dca6ba46 google::(anonymous namespace)::FailureSignalHandler() at ??:0
W20260619 08:55:01.995440 1228 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44186: Illegal state: replica 408d1f4ba9f245f1aa2608445e067a01 is not leader of this config: current role FOLLOWER
@ 0x7f32de835980 (unknown) at ??:0
@ 0x7f32da583e87 gsignal at ??:0
@ 0x7f32da5857f1 abort at ??:0
@ 0x7f32dca5d9ad google::LogMessage::Fail() at ??:0
@ 0x7f32dca6088a google::LogMessage::SendToLog() at ??:0
@ 0x7f32dca5d4de google::LogMessage::Flush() at ??:0
@ 0x7f32dca5ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x43b95e kudu::tserver::RaftConsensusITest::StopOrKillLeaderAndElectNewOne() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/raft_consensus-itest.cc:435 (discriminator 21)
@ 0x446ccf kudu::tserver::RaftConsensusITest_MultiThreadedInsertWithFailovers_Test::TestBody() at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/raft_consensus-itest.cc:1060
@ 0x7f32dde77bfe testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f32dde6c5db testing::Test::Run() at ??:0
@ 0x7f32dde6c795 testing::TestInfo::Run() at ??:0
W20260619 08:55:02.004634 1358 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54438: Illegal state: replica 4e0e4dfff5e94d4b8e4b97882c68c5a9 is not leader of this config: current role FOLLOWER
W20260619 08:55:02.004725 1359 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54438: Illegal state: replica 4e0e4dfff5e94d4b8e4b97882c68c5a9 is not leader of this config: current role FOLLOWER
W20260619 08:55:02.004634 1360 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54438: Illegal state: replica 4e0e4dfff5e94d4b8e4b97882c68c5a9 is not leader of this config: current role FOLLOWER
W20260619 08:55:02.004837 1357 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54438: Illegal state: replica 4e0e4dfff5e94d4b8e4b97882c68c5a9 is not leader of this config: current role FOLLOWER
W20260619 08:55:02.004932 1356 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54438: Illegal state: replica 4e0e4dfff5e94d4b8e4b97882c68c5a9 is not leader of this config: current role FOLLOWER
W20260619 08:55:02.005575 1356 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54438: Illegal state: replica 4e0e4dfff5e94d4b8e4b97882c68c5a9 is not leader of this config: current role FOLLOWER
@ 0x7f32dde6ccdf testing::TestSuite::Run() at ??:0
@ 0x7f32dde6d29f testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7f32dde7813e testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f32dde6c83d testing::UnitTest::Run() at ??:0
@ 0x438cba main at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_main.cc:110
@ 0x7f32da566c87 __libc_start_main at ??:0
@ 0x438ef2 (unknown) at ??:?